Location resolution of social media posts

ABSTRACT

The present disclosure relates generally to internet social media, and more specifically to techniques for determining location-related information about internet social media content. In some embodiments, a system accesses data representing a first social media post, the data including geographic location data identifying a first geographic location. The system identifies a second social media post related to the first post. The system accesses data representing the second social media post, wherein the data representing the second post does not include geographic location data identifying the first geographic location. The system analyzes the data representing the second social media post and determines a location score based at least in part on the analysis of the data representing the second social media post. If the location score exceeds a threshold location score, the system associates the second social media post with the first geographic location.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/424,707, filed Feb. 3, 2017, which claims priority to U.S.Provisional Application Ser. No. 62/403,618 filed Oct. 3, 2016. All ofthese applications are incorporated by reference herein in theirentirety.

FIELD

The present disclosure relates generally to internet social media, andmore specifically to techniques for determining location-relatedinformation about internet social media content.

BACKGROUND

Social media platforms are becoming an increasingly important way tointeract and engage with people, such as customers of a business. Bothcustomers and businesses can benefit from an increase in the number ofopportunities to engage through the use of social media. In particular,the benefits of engaging are particularly pronounced when a customerattends an event at the physical, geographic location of a business orevent. The ability to recognize the customer's physical presence andengage with the customer in real time, can be used to enhance thecustomer's experience at the location, and thus is extremely valuable.

However, some techniques for determining the presence of a customer at aphysical location using social media are extremely limited in theirability to effectively determine relevant location information,particularly in real time. For example, some existing techniques relysolely on social media posts with geotagged physical locations in orderto determine social media activity by customers that are physicallypresent at a location. However, techniques that rely solely on geotaggedsocial media posts can miss the vast majority (e.g., up to about 95%) ofsocial media activity relating to the geographic location of interest.

BRIEF SUMMARY

The present techniques provide methods, computer-readable media, andsystems for effectively determining location-related information aboutinternet social media content.

In some embodiments, a computer-implemented method for resolvinggeographic location of social media posts comprises: accessing, by oneor more processors, data representing a first social media post, whereinthe data representing the first social media post includes geographiclocation data identifying a first geographic location; identifying, byone or more processors, a second social media post related to the firstsocial media post; accessing, by one or more processors, datarepresenting the second social media post, wherein the data representingthe second social media post does not include geographic location dataidentifying the first geographic location; analyzing, by one or moreprocessors, the data representing the second social media post;determining, by one or more processors, a location score based at leastin part on the analysis of the data representing the second social mediapost; and if the location score for the data representing the secondsocial media post exceeds a threshold location score, associating thesecond social media post with the first geographic location.

In some embodiments, a non-transitory computer-readable storage mediumstores one or more programs, the one or more programs comprisinginstructions, which, when executed by one or more processors of asystem, cause the system to: access data representing a first socialmedia post, wherein the data representing the first social media postincludes geographic location data identifying a first geographiclocation; identify a second social media post related to the firstsocial media post; access data representing the second social mediapost, wherein the data representing the second social media post doesnot include geographic location data identifying the first geographiclocation; analyze the data representing the second social media post;determine a location score based at least in part on the analysis of thedata representing the second social media post; and if the locationscore for the data representing the second social media post exceeds athreshold location score, associate the second social media post withthe first geographic location.

In some embodiments, a system comprises: one or more processors; memory;and one or more programs, wherein the one or more programs are stored inthe memory and configured to be executed by the one or more processors,the one or more programs including instructions for: accessing datarepresenting a first social media post, wherein the data representingthe first social media post includes geographic location dataidentifying a first geographic location; identifying a second socialmedia post related to the first social media post; accessing datarepresenting the second social media post, wherein the data representingthe second social media post does not include geographic location dataidentifying the first geographic location; analyzing the datarepresenting the second social media post; determining a location scorebased at least in part on the analysis of the data representing thesecond social media post; and if the location score for the datarepresenting the second social media post exceeds a threshold locationscore, associating the second social media post with the firstgeographic location.

DESCRIPTION OF THE FIGURES

FIG. 1 depicts a network diagram in accordance with some embodiments.

FIG. 2 depicts a flow diagram illustrating an exemplary process fordetermining, accessing, and analyzing social media posts in accordancewith some embodiments.

FIG. 3 depicts an exemplary dataset representing a plurality of socialmedia posts in accordance with some embodiments.

FIG. 4 depicts exemplary interfaces for displaying social media posts inaccordance with some embodiments.

FIG. 5 depicts images associated with social media posts in accordancewith some embodiments.

FIG. 6 depicts an exemplary dataset representing a plurality of socialmedia posts in accordance with some embodiments.

FIG. 7 depicts exemplary interfaces for displaying social media posts inaccordance with some embodiments.

FIGS. 8A-8B depict images associated with social media posts inaccordance with some embodiments.

FIG. 9 depicts a flow diagram illustrating an exemplary process foranalyzing images associated with social media posts in accordance withsome embodiments.

FIG. 10 depicts a flow diagram illustrating an exemplary process foranalyzing images associated with social media posts in accordance withsome embodiments.

FIG. 11 depicts a flow diagram illustrating an exemplary process foranalyzing images associated with social media posts in accordance withsome embodiments.

FIG. 12 depicts a flow diagram illustrating an exemplary process foranalyzing images associated with social media posts in accordance withsome embodiments.

FIG. 13 depicts a flow diagram illustrating an exemplary process foranalyzing images associated with social media posts in accordance withsome embodiments.

FIG. 14 depicts a flow diagram illustrating an exemplary process foranalyzing images associated with social media posts in accordance withsome embodiments.

FIG. 15 depicts a flow diagram illustrating an exemplary process foranalyzing text associated with social media posts in accordance withsome embodiments.

FIG. 16 depicts a flow diagram illustrating an exemplary process foranalyzing text associated with social media posts in accordance withsome embodiments.

FIG. 17 depicts a flow diagram illustrating an exemplary process fordetermining a geographic location of a social media post in accordancewith some embodiments.

FIG. 18 depicts a flow diagram illustrating an exemplary process fordetermining a geographic location of a social media post in accordancewith some embodiments.

FIG. 19 depicts a flow diagram illustrating an exemplary process fordetermining a location score for social media posts in accordance withsome embodiments.

FIG. 20 depicts a flow diagram illustrating an exemplary process fordetermining a return on engagement in accordance with some embodiments.

FIG. 21 depicts a flow diagram illustrating an exemplary process fordetermining a social activity index in accordance with some embodiments.

FIG. 22 depicts a flow diagram illustrating an exemplary process fordetermining a social influence index in accordance with someembodiments.

FIG. 23 depicts a flow diagram illustrating an exemplary process foraccessing data regarding a geographic location in accordance with someembodiments.

FIG. 24 depicts a flow diagram illustrating an exemplary process foraccessing data regarding a geographic location in accordance with someembodiments.

FIGS. 25A-25G depict interfaces for displaying geographic locationinformation associated with social media posts in accordance with someembodiments.

FIG. 26 illustrates a functional block diagram of a computing system inaccordance with some embodiments.

DETAILED DESCRIPTION

The following description is presented to enable a person of ordinaryskill in the art to make and use the various embodiments. Descriptionsof specific devices, techniques, and applications are provided only asexamples. Various modifications to the examples described herein will bereadily apparent to those of ordinary skill in the art, and the generalprinciples defined herein may be applied to other examples andapplications without departing from the spirit and scope of the variousembodiments. Thus, the various embodiments are not intended to belimited to the examples described herein and shown, but are to beaccorded the scope consistent with the claims.

FIG. 1 depicts an exemplary network 100, utilized in accordance withsome embodiments. As depicted in FIG. 1, in some examples, user devices104, social media servers 106, location resolution system 108, and venuecomputer system 110 are each connected to data network 102. Data network102, for example, can be any suitable data network for connectingcomputing devices and/or systems. A data network as used herein is, forexample, a wide area network (“WAN”) (e.g., the Internet), a local areanetwork (“LAN”), or the like, or some combination thereof. One of skillin the art will readily appreciate that data network 102 is ageneralized depiction, and that the data communication channel betweendevices and systems 104, 106, 108, and 110 can be comprised of one ormore interconnected networks.

FIG. 2 depicts a flow chart of an exemplary process 200 for identifyingand analyzing a public social media post, in accordance with someembodiments. At box 202, a system (e.g., location resolution system 108of FIG. 1) accesses a public social media post from a social medianetwork via an application programming interface. For example, withreference to FIG. 1, a user of one of user devices 104 creates, via datanetwork 102, a social media post on a social media network. In thisexample, the social media platform is represented by social mediaservers 106, which host the social media network. A social media networkcan also be referred to as a “social media platform”, “social mediawebsite”, “social media provider”, “social media service”, “social mediamessage board”, or the like. Exemplary social media networks include,for example, well-known networks such as: Facebook (by Facebook, Inc. ofMenlo Park, Calif., US), Instagram (by Facebook, Inc. of Menlo Park,Calif., US), and Twitter (by Twitter, Inc. of San Francisco, Calif.,US). The phrase “social media network” is not intended to limit thescope of the embodiments described herein, and can also refer to anycomputerized system or network that can be used to create, share,exchange, and view user-generated content. User-generated and/oruser-shared content on a social media network is referred to hereinafteras a “social media post” or simply a “post”. In some examples, a socialmedia network utilized in accordance with techniques described herein,has the following characteristics: it allows publically-viewable posts,it allows user accounts to post images and/or video in addition to text,and it makes content accessible via an application programming interfaceor other method that allows third-party systems to access and process astream of content from the social media network.

In some embodiments, a system (e.g., location resolution system 108 ofFIG. 1) accesses content on a social media network using an applicationprogramming interface (“API”). An API, as is well-known in the art,provides an interface standard that allows an application (e.g.,executing on location resolution system 108 of FIG. 1) to communicatewith and access information from a social media network (e.g., on socialmedia servers 106 of FIG. 1). An API can be unique to a particularsocial media network, or can be a common API utilized by a plurality ofsocial media networks. The operation of APIs is well understood by thoseof skill in the art, and thus is not discussed in further detail.

At box 204, the system analyzes the social media post. For example, aswill be discussed in further detail below, the system (e.g., locationresolution system 108 of FIG. 1) analyzes the content of a social mediapost to determine a geographic location associated with the post. At box206, the system outputs the results of the analysis.

1. Initial Data Processing of Social Media Posts Having Known GeographicLocations

The techniques described below are useful, for example, for determininga geographic location of a social media post that has not beengeotagged. As described above, non-geotagged social media posts accountfor a large proportion of posts that would otherwise be missed by abusiness (e.g., an event venue or operator), which represents a lostopportunity to view customer feedback and sentiment, and to engage withcustomers through social media. On the other hand, by generating thisotherwise non-existent location data, more robust and actionableanalytic data can be created and used by businesses.

Turning to FIG. 3, table 300 (also referred to as dataset 300) depictsdata representing a plurality of social media posts that each includes ageotag. A geotag can be a particular location (e.g., venue), latitudeand longitude coordinates, or the like. Because the posts of table 300each include a geotag, they are considered social media posts having aknown geographic location, or simply, location. In some embodiments, asocial media post without a geotag can have a known location, forexample, when such post has been previously analyzed (and locationdetermined), or when user input has provided and/or confirmed thelocation of the social media post (e.g., user input following visualconfirmation by a user that an image depicts the geographic location),or when the location was otherwise previously-associated with the socialmedia post (e.g., contained in metadata of an image), or any othersituation in which explicit location data of a social media post isassociated with the social media post's data.

In some embodiments, a system in accordance with the techniquesdescribed herein accesses, creates, or otherwise uses a dataset ofsocial media posts known to have been posted at a geographic locationand/or that depict a geographic location (e.g., are geotagged at aparticular venue, or that depicts a venue). As described above, in someembodiments, the dataset includes data from one or more of: geotaggedsocial media posts, and social media posts confirmed (e.g., by analysisor by user input) to have been posted at and/or that depict thegeographic location.

Social media posts can typically be represented as several fields ofdata representing the information from the post. In some embodiments, adataset representing one or more social media posts includes, for eachsocial media post, one or more of the following fields: text,image/video, and hashtag. In some embodiments, each social media postdoes not require all three pieces of information to be stored in adataset. For example, post number 4 in FIG. 3 includes text andhashtags, but does not include an image or image identifier reference.Thus, the appropriate row and column does not include reference to animage associated with the post.

In some embodiments, other data fields can be included in a datasetrepresenting a social media post, including one or more of thefollowing: a user account, a user name, date, time, geotag, informationregarding comments/responses for a post (e.g., the text of a comment, anidentifier of the user account that posted the comment, and the like),user interactions with the post (e.g., number of “likes”, “shares”, orother indications that a user account interacted with the post),information about the user device used to create the post, and otherappropriate data that can be associated with the social media post.

In some embodiments, the dataset is stored as a database. In thisexample, the dataset represented by table 300 is stored in one or moredatabase files (e.g., on location resolution system 108 of FIG. 1). Adataset can be stored in any appropriate format for storing andaccessing data using computer-accessible media, including one or more ofthe following formats: Extensible Markup Language (XML), comma-separatedvalues (CSV), JavaScript Object Notation (JSON), Structured QueryLanguage (SQL), Hierarchical Data Format (HDF), and plain text. Those ofskill in the art will recognize that this list is not exhaustive, andthat other data formats can be used instead or in addition to thoselisted here. As described below, the dataset can be used to create ortrain classifier processes for determining a location of social mediaposts that do not include an explicit association with a location.

The data included in a dataset, such as the dataset represented by table300 of FIG. 3, can be collected from one or more sources. In someembodiments, the data is retrieved from a social media network (e.g.,retrieved by location resolution system 108 and/or venue computer system110 from social media servers 106 of FIG. 1). For example, the venuecomputing system can interface with social media network servers via anAPI and retrieve data representing social media posts that arepublically-viewable (e.g., viewable by the general public, or any userwith an account on a social media network; these are otherwise referredto as “public” posts). Other techniques for retrieving relevant data arecontemplated, such as the use of data scraping (e.g., web scraping), orby capturing data via user input. In some embodiments, the social mediaposts represented by the data are public posts.

Table 300 includes data representing four social media posts (the postsnumbered 1 through 4). Table 300 (otherwise referred to as “dataset 300”or “initial dataset 300”) includes a post identifier field 302, a timefield 304, a text field 306, a hashtag field 308, an image identifierfield 310, and a geotag field 312—each of which includes informationcorresponding to a respective post (where each row in table 300represents a post). For instance, the data representing post 1 of table300 includes the following information associated with the post: timeand date of the post (Aug. 13, 2018 at 7:22 PM), the text content of thepost (“What a beautiful new arena! Can't wait to watch Team playtonight!”), hashtags from the post (# CityArena, # Gametime, and #GoTeam), an identifier for an image included in the post (image101.jpg),and geotag information (geotagged at City Arena). Posts 2 through 4include the same information fields, each populated by respectivecontent. Notably, post 4 does not include an image, so its respectiveimage identifier field 310 does not contain an image identifier. Thedata in table 300 can be stored in accordance with the techniquesdescribed above.

In some embodiments, the data is stored by venue. For example, data froma single known geographic location (e.g., City Arena, a venue) is storedtogether as a dataset for that specific venue. In some embodiments, thedata is stored by venue type. For example, data from a plurality ofvenues can be stored together as a dataset for a venue type. Forinstance, a dataset can include images from several differentprofessional basketball arenas. Because each basketball arena in thedataset has generally similar common features, this dataset can beuseful for identifying other basketball arenas.

FIG. 4 depicts exemplary interfaces for displaying the social mediaposts represented in table 300 of FIG. 3. For example, interface 402depicts a visual arrangement of the data representing post 1 in table300 that can be displayed when a user accesses a social media networkfrom a user device, such as a personal computer or a smartphone (e.g.,one of user devices 104 of FIG. 1). Similarly, interfaces 404 through408 depict visual arrangements of posts 2 through 4 of table 300,respectively. The interfaces of FIG. 4 are provided as an illustrativeexample of a social media posts. However, as one skilled in the artwould appreciate, social media posts corresponding to any number ofinterfaces and/or arrangements are contemplated to be within the scopeof the techniques described herein.

FIG. 5 depicts exemplary images associated with social media posts. Inthis example, the images 502, 504, and 506 correspond, respectively, tosocial media posts 1, 2, and 3 of table 300 of FIG. 3. Image 502corresponds to the image identifier “image101.jpg”, image 504corresponds to the image identifier “image102.jpg”, and image 506corresponds to the image identifier “image103.jpg”, each listed in table300. In the examples depicted, the images were each taken shortly beforeand during a professional basketball game, taken just outside and insideof a basketball arena venue (e.g., “City Arena”).

As explained in greater detail below, a dataset (e.g., dataset 300) thatincludes information about social media posts (or other relevantinformation) having a known geographic location can be used to determinea geographic location of social media posts having no known locationdata.

Turning to FIG. 6, table 600 depicts data representing a plurality ofsocial media posts, the majority of which do not include a geotag. Theposts of table 600 that do not include a geotag (e.g., posts 1 through 3and posts 5 through 8) are considered social media posts that do nothave a known geographic location. Similar to table 300 of FIG. 3, table600 (otherwise referred to as “dataset 600”) includes a post identifierfield 602, a time field 604, a text field 606, a hashtag field 608, animage identifier field 610, and a geotag field 612—each of whichincludes information corresponding to a respective post (where each rowin table 600 represents a post). For instance, the data representingpost 1 of table 600 includes the following information associated withthe post: time and date of the post (Aug. 10, 2018 at 12:47 PM), thetext content of the post (“Check out this new band”), and hashtags fromthe post (# newband), but does not include information in the respectiveimage identifier or geotag fields. Posts 2 through 8 include the sameinformation fields, each populated by respective content. Notably, post4 includes a geotag (geotagged at City Arena). In this example, post 4of table 600 corresponds to post 2 from table 300. The data in table 600can be stored in accordance with the techniques described above. Adataset (e.g., tables 300, 600) can include all of the data describedabove, less than all of the data described above, or more than the datadescribed above.

In this example, each of the social media posts represented by the rowsof table 600 was posted by a single user. The only post that includes ageotag is post 4; the remaining posts do not include a geotag or otherexplicit identification of location. While it is clear that post 4,based on its associated geotag and time, was posted from the basketballgame that began at 7:30 PM at City Arena on Aug. 13, 2018 (e.g.,discussed above with reference to FIG. 5), the remaining posts of table600 also contain potentially valuable information that would otherwiseremain undiscovered by merely searching social media based on the geotag“City Arena”. For example, posts 2, 3, and 5-8 of table 600 are relatedto the basketball game, and posts 3 and 5-8 were posted during thebasketball game (or otherwise close in time to the game), but do notinclude a geotag (e.g., “City Arena”). Accordingly, the non-geotaggedsocial media posts 1-3 and 5-8 of table 600 are excellent candidates forlocation resolution analysis. Techniques for such analysis are describedin greater detail below.

FIG. 7 depicts exemplary interfaces for displaying the social mediaposts represented in table 600 of FIG. 6. For example, interface 702depicts a visual arrangement of the data representing post 1 in table600 that can be displayed when a user accesses the social media networkfrom a user device, such as a personal computer or a smartphone (e.g.,one of user devices 104 of FIG. 1). Similarly, interfaces 704 through716 depict visual arrangements of posts 2 through 8 of table 600,respectively. The interfaces of FIG. 7 are provided as an illustrativeexample of a social media posts. However, as one skilled in the artwould appreciate, social media posts corresponding to any number ofinterfaces and/or arrangements are contemplated to be within the scopeof the techniques described herein.

FIGS. 8A-8B depict exemplary images associated with social media posts.In this example, the images 802 and 804 of FIG. 8A, and images 806 and808 of FIG. 8B correspond, respectively, to social media posts 3, 4, 6,and 8 of table 600 of FIG. 6. Image 802 corresponds to the imageidentifier “image201.jpg”, image 804 corresponds to the image identifier“image202.jpg”, image 806 corresponds to the image identifier“image203.jpg”, and image 808 corresponds to the image identifier“image204.jpg”, each of table 600. In the example depicted, the imageswere each taken shortly before or during the professional basketballgame, and taken just outside or inside of a basketball arena (e.g.,“City Arena”).

2. Classification and Analysis of Social Media Post Data

Determining a geographic location of a social media post that lacks ageotag or other explicit indication of location in the data representingthe social media post can involve analysis of the underlying datarepresenting that post. In some embodiments, the technique includes theuse of image classifiers to analyze an image (and/or video) associatedwith a social media post. In some embodiments, the technique includesthe use of semantic analysis of the text and/or hashtags associated witha social media post. In some embodiments, a location resolution systemdetermines a geographic location based on one or more of results ofimage and semantic analysis. These aspects are discussed below, in turn.

a. Image Classifiers

In some embodiments, if a social media post includes an image, the imageis analyzed using one or more image classifier process. In someexamples, the image classifiers are computer-implemented processes usedto classify an image into one or more “classes” or with one or more“labels”. In some embodiments, the classification of an image involvesthe analysis of the visual content of the image. The analysis of theimage can be used to label the image, and determine, based on the label,a geographic location to be associated with the social media post thatincluded the image.

An object that is configured to classify an image or video is referredto as a “classifier” or a “classifier object”. In some embodiments, aclassifier is a computer-executable program, object, routine, function,process, and/or some other set of computer-executable instructions. Theterm “image” as used throughout this document refers to either, or both,an image (e.g., a photograph) or a video (e.g., a collection of imageframes), unless otherwise noted.

Exemplary techniques and processes for performing image classificationin accordance with some embodiments are discussed below.

i. k-Nearest Neighbors Image Classifiers

In some embodiments, an image classifier utilizes a Nearest Neighborsalgorithm (also referred to as a “k-Nearest Neighbors algorithm” or“k-NN algorithm”). A k-NN algorithm can be used to classify a targetobject (e.g., determine a class membership of an image or video) basedon the k number of nearest feature vectors in a given feature space oftraining samples for which classes are known. In some examples, if k isequal to one, then the algorithm can assign (to the target object) theclass label of the object whose feature vector is nearest to the featurevector of the target object being classified. In some examples, if k>1,then a majority vote of the k nearest feature vectors can be used toassign a class label to the target object being classified.

Application of a classifier that utilizes a k-NN classification schemegenerally involves two processes: “building” the classifier (which canalso be referred to as “training” the classifier), and performingclassification of target objects using the classifier.

FIG. 9 depicts an exemplary flow diagram of a process 900 for building aclassifier that utilizes a k-Nearest Neighbors classification scheme. Atblock 902, a system ingests a set of images. In some embodiments,ingesting data includes parsing text associated with the data. In someembodiments, ingesting a set of images can include accessing alocally-stored set of images and/or retrieving images from one or moreremote locations (e.g., from one or more social media networks via theInternet). For example, the system can access the images (depicted inFIG. 5) of the posts in dataset 300 of FIG. 3, which can be stored withdata of dataset 300 or retrieved using an identifier or path in the dataof dataset 300 (e.g., an Internet URL, a file directory location). Inthis example, a classifier built using the images from dataset 300 willresult in a useful tool for classifying social media posts of unknownlocation that include images, because the posts of initial dataset 300are each geotagged (and thus each image has a known location associatedwith it).

In some embodiments, the images of the set of images are associated witha set of labels (also referred to as “classes”). For example, the imagesof a set of images (associated with a known location) can each belabeled (e.g., associated with a label) based on what is visuallydepicted in the images. Ingesting a set of images can further includedetermining the number of images in the set of images and the number andidentity of any labels associated with each image. For example, if 100images are ingested, then m_(images)=100. If the 100 images include 3labels, n_(labels)=3 (e.g., a set of labels such as {“playfield”,“selfie”, and “other”}). In this example, the three images of dataset300 (e.g., shown in FIG. 5) can be considered the first 3 images of the100 images ingested.

In some embodiments, ingesting a set of images includes retrieving theimages and their corresponding labels. For example, the images and theirlabels can be included in a pre-existing set of data that is retrievedfrom computer memory.

At block 904, the system pre-processes the set of images. In someembodiments, pre-pre-processing includes converting each image into apredefined shape. For example, the system can convert each image into a100×100 pixel image. In some embodiments, pre-processing includesadjusting and/or normalizing one or more visual characteristics ofimages in the set of images. For example, the system can adjust and/ornormalize image color, brightness, or the like.

At block 906, the system creates a feature vector. In some embodiments,a feature vector is created for each image in the set of images. Afeature vector (or other data structure such as an array) represents animage based on one or more features that characterize the image. Featurevectors can be created which quantitatively express features of imagesin a way that makes comparison easier. Features useful for the analysisof images can include, but are not limited to: pixel colors,distribution of colors among patches of pixels in the image, imagehistogram, color histogram, or the like.

In some embodiments, creating a feature vector includes, for each imagein the set of images, creating a vector for each channel of the image.For example, for an RGB image (an image whose data includes red, green,and blue color channels), a vector is created by flattening the matrixcorresponding to each color channel. In this example, three vectors willbe created—a vector for each of the red, green, and blue channels. Thus,each channel vector of a 100×100 pixel RGB image will be 10,000 elementslong (representing each pixel in the 100×100 pixel image).

In some embodiments, the channel vectors are concatenated. In thisexample, the concatenated vector would be 30,000 elements long(10000+10000+10000)—the concatenation of the three RGB color channelvectors. In some embodiments, further processing is performed using thechannel vectors and/or the concatenated vector. Further processing canbe performed, for example, to determine the feature vectors of eachimage. Process 1000, described below, illustrates exemplary furtherprocessing for determining feature vectors.

At block 908, the system stores the feature vector. For example, onceeach image has been processed and represented as one or more featurevectors (or arrays), the system stores the feature vector information,which can be used in the analysis of images of unknown class (e.g., bycomparing a feature vector of the image of unknown class with the storeddata).

FIG. 10 depicts an exemplary flow diagram of process 1000 for processingimage channel vector data. At block 1002, the system creates sparsematrix recommender (“SMR”) sub-matrices from vector representations ofthe images. In some embodiments, the system creates a sub-matrix foreach color channel of the image. Depending on the color channels used,these sub-matrices can be referred to as Red-Green-Blue (RGB) matrices,Hue-Saturation-Value (HSV) matrices, or the like. HSV matrices can beused in conjunction with, or instead of, RGB matrices. For example, ifeach image in the set of images is resized to be 100×100 pixels, thesystem can create a sub-matrix that includes the red color channelinformation for all of the images (of the set of images), and that hasthe dimensions 100×10000 (100 images (rows), each having 10,000 pixels(columns)). Similarly, a sub-matrix for the green pixels with the samedimensions, and a sub-matrix for the blue pixels of the same dimensionscan be created.

At block 1004, the system creates quantized value matrices from thesub-matrices. For example, the values of the pixels in each colorchannel that range from 0 to 255 (e.g., for an 8-bit color channel) areseparated into 10 intervals. For instance, the range of values from 0 to255 is divided into 10 intervals, and a value in the range isrepresented by one of the 10 intervals (e.g., with the first intervalrepresenting the values from 0 to 25, the second interval representingthe values from 26-50, and so on). Thus, in this example, a value of 13would have the quantized value of 1 (e.g., because it belongs to thefirst interval), and a value of 29 would have a quantized value of 2(e.g., because it belongs to the second interval).

At block 1006, the system creates incidence matrices for each pixelposition and quantized value. For example, this can be explained asunfolding each column into quantized values columns. For instance, ifthe pixel at position (e.g., column) 125 in the 37th vectorized image(e.g., row) (of the 100 images) has the quantized value of 6, then thesystem replaces the column 125 with 10 columns and assigns the value of1 to the 6th column of those new columns. In absolute coordinates thismeans that the matrix entry (37, 124*10+6) will have the value 1. Inother words, the system expands a matrix entry that includes a (single)quantized value into a number of entries equal to the number ofquantization intervals (e.g., 10 in this example), where the quantizedvalue is represented by the position of an entry (e.g., non-zero entry).Here, the quantized value of [6] can be represented as [0, 0, 0, 0, 0,1, 0, 0, 0, 0]. Thus, in the case of 10 quantization intervals, a100×10,000 color channel sub-matrix would expand to have the dimensions100×100,000. Each row of the incidence matrix can be used to compute thesimilarity between images. For example, the more non-zero values incommon that two rows have (e.g., each row representing a differentimage) the more similar the images are. In the example above, the matrixentry (37, 124*10+6) has a value of 1—if the matrix entry (100,124*10+6) also has the value of 1 this means that the 37th image and100th image have the same (quantized) color value at the correspondingpixel. The greater the number of pixels that have the same color values,the more similar the images.

At block 1008, the system creates a sub-matrix for a set of labels. Insome embodiments, each column corresponds to a label. For example, theset of labels can be {“playfield”, “selfie”, “other”}. In this example,an image associated with the label “playfield” would depict a basketballcourt. In other examples, an image labeled “playfield” can depict aplaying field, playing surface, court, or other space in which asporting event is played. Images of the set that are labeled “playfield”can include images of the same playfield or different playfields. Insome examples, an image labeled “playfield” can also depict the stadium,venue, and/or building (e.g., with or without including a playingsurface). Such images taken at the physical, geographical location of anevent, such as a sporting event, tend to indicate that the user is (orwas) physically present at the event.

In some embodiments, the resulting sub-matrix for the set of labels hasthe dimensions n_(images)×n_(labels), wherein n_(labels) is the numberof labels in the set of labels. In this example, the sub-matrix oflabels has the dimensions 100×3 (100 rows and 3 columns). Thus, each rowcorresponds to an image of the set of 100 images, and each columncorresponds to a label of the set of 3 labels. For ease of reference,the sub-matrix for the set of labels will be referred to as M_(lbl). Insome embodiments, the sub-matrix for the set of labels includes a firstvalue (e.g., “1”) at entries that correspond to images that areassociated with a label, and a second value (e.g., “0”) at entries thatcorrespond to images that are not associated with the label. Forexample, if the image represented by row 1 of the sub-matrix is labeled“playfield” from the set of labels {“playfield”, “selfie”, “other”}, thefirst column of row 1 includes a value of 1, but the second and thirdcolumns include a value of 0.

At block 1010, the system splices the sub-matrices into one matrix. Forexample, the channel sub-matrices are spliced into one matrix. In someexamples, the incidence matrices created from the channel sub-matricesare spliced with the label sub-matrix to form a single matrix. In thisexample, the spliced matrix would have the dimensions 100×300,003. Thatis, for the set of 100 images (100 rows), with each 100×100 pixel imagehaving 3 color channels (e.g., a red, green, and blue channel) quantizedinto 10 intervals, and the images categorized according to three labels:(100 pixels)*(100 pixels)*(3 channels)*(10 intervals)+(3 labels)=300,003columns. For ease of reference, the spliced matrix will be referred toas M. As can be seen, in this example the spliced matrix M includes thelabel sub-matrix M_(lbl). In some embodiments, a spliced matrix does notinclude a label matrix. For instance, if the label sub-matrix M_(lbl) isexcluded in this example, M would have the dimensions 100×300,000.

At block 1012, the system performs a latent semantic indexing processusing the entries of selected sub-matrices. In some embodiments, latentsematic indexing (“LSI”) is performed on one or more of the sub-matricesthat are included in the spliced matrix M. In some examples, LSI isperformed on one or more of the incidence matrices created from thechannel sub-matrices. In some embodiments, latent sematic indexing isperformed on the sub-matrices of M that are not label sub-matrices(e.g., LSI is performed on the RGB color channel sub-matrices, but noton the label sub-matrix M_(lbl). In some embodiments, LSI is optionallyperformed. For example, LSI can be performed sometimes, or not at all,to the sub-matrices representing a set of images. Latent semanticindexing is a technique that utilizes statistics about the appearance ofmatrix values across the matrix columns and rows. For example, the LSIprocess is used to determine patterns within an individual color channelsub-matrix (e.g., an incidence matrix for the color channel) for imagesof the set of images that share a label. Thus, the system can determinethe latent patterns that are present in the data of images that areknown to be labeled, for example, as “playfield”. For instance,referring back to FIG. 5, images 504 and 506 each depict the basketballcourt of the venue City Arena, but from different angles and withdifferent fields of view. Though the images are not identical (or evenvery similar), because they depict the same subject matter they willhave some latent similarities in their visual data (e.g., the pixelcoloration). Using an LSI process allows those similarities to beidentified and quantified.

In some embodiments, one or more of the following LSI metrics are used:inverse document frequency (“IDF”), global frequency inverse documentfrequency (“GFIDF”), and term frequency-inverse document frequency(“TF-IDF”). One skilled in the art would appreciate that other LSImetrics not listed here can be used instead of, or in addition to, theaforementioned metrics.

In some embodiments, one or more sub-matrix row is normalized. In someexamples, applying LSI includes applying normalization to eachsub-matrix row. For example, a cosine normalization function can beapplied to each row of a sub-matrix, wherein each row represents avector. Cosine normalization is applied to each row i of the sub-matrixM using the formula: (M[i,j])/sqrt(M[i,1]²+M[i,2]²+ . . .M[i,n_(normal)]²), wherein “sqrt( )” represents the square root of theexpression inside the parentheses, wherein M[i,j] is the entry locatedat row i and column j of matrix M (having m rows and n columns), andwherein n_(normal) is the number of columns (less than or equal to n) tobe normalized. For example, if 30,000 columns of M are being normalized,then n_(normal)=30000. In some examples, fewer than all of the columns nare normalized (e.g., if M includes n=30,003 columns, n_(normal) can be30000). One skilled in the art would appreciate that other normalizationfunctions can be used. Normalizing the sub-matrices is an importantfeature for tuning the classifier being built.

In some embodiments, one or more of the sub-matrix M_(lbl), the matrixM, the normalized sub-matrices, the quantized value matrices, theincidence matrices, and the results of the latent semantic indexing arestored in computer memory. In some embodiments, one or more of thesematrices are stored as an object in computer memory. This can bereferred to as an “SMR object” or a “classifier object”.

FIG. 11 depicts an exemplary process 1100 for classifying a target imageusing a classifier that utilizes a k-NN classification scheme. In someexamples, this process includes using a classifier object (e.g., an SMRobject) to classify an inputted image, wherein the output is a list ofpairs, each pair being a label and an associated confidence level.

At box 1102, a system receives an image. For example, the systemreceives the target image, identified as “image201.jpeg” of dataset 600(e.g., image 802 of FIG. 8). In some embodiments, the image (or anaddress or other identifier of the image) is retrieved via an API of asocial media network as described above. The received image may bepre-processed as described above.

At box 1104, the system calculates a feature vector of the image. Forexample, the image is transformed into a vector v_(i) as described abovewith regard to blocks 904 and 906 of process 900 (FIG. 9.) For instance,the image can be pre-processed and converted into a predefined shape(e.g., 100×100 pixels), flattened into three vectors representing RGBcolor channels (three vectors of 10,000 elements in length), andconcatenated (into a single 30,000 element vector). In some examples,calculating the feature vector includes one or more of the followingprocesses described above with respect to process 1000 (FIG. 10):creating SMR sub-matrices from a vector representation of the image,creating quantized value matrices from the sub-matrices, and creatingincidence matrices for each pixel position and quantized value. In someexamples, one or more other appropriate pre-processing techniques can beused, such as image key points, image patch color histograms, or thelike.

At box 1106, the system determines the top-k number of nearest neighborsof the received image's feature vector. In some embodiments,matrix-vector multiplication is performed in order to determine thetop-k nearest neighbors (e.g., feature vectors). For example, thefollowing equation can be used: s=(M)·(v_(i))^(T), where s is a vectorcomposed of the dot product of the transposed feature vector v_(i) witheach vector of the initial dataset (e.g., that make up matrix M). Inthis example, if M is 100×300,003 and v_(i) is 1×300,003, then s is100×1. The entries of the resulting vectors correspond to similarityscores computed with the sparse matrix recommender. A vector s_(k) canthen be created which includes non-zero entries only for the top-kentries (e.g., rows of s).

At box 1108, the system determines the labels associated with the top-knearest neighbors. This can also be referred to as determining thelabel-confidence pairs. In some embodiments, the label-confidence pairsare determined by multiplying the vector s_(k) by the sub-matrix oflabels M_(lbl). For example, the pairs are given by the vector result,v_(result), of the following expression: v_(result)=v/sum(v), whereinv=(s_(k))·(M_(lbl)).

At box 1110, the system determines a label for the received image. Insome embodiments, the system determines a plurality of labels for thereceived image. For example, an image may have more than one labelassociated with it. For instance, if an image depicts both a person anda playing field, it can be labeled both “playfield” and “selfie”.

There are multiple techniques for determining the label of an image, forexample, where the top k nearest neighbors correspond to multiplelabels.

In some embodiments, the number of occurrences of each label of thetop-k nearest neighbors are added up. This is referred to as a simplevoting scheme. For instance, if k=3 and two of the nearest neighbors areeach only associated with the label “playfield” and one of the nearestneighbors is only associated with the label “selfie”, then the receivedimage is labeled as “playfield” (e.g., by 2 votes to 1). In other words,the received image is classified with the label whose feature vectorsare most frequent among the top-k most similar feature vectors to thefeature vector v_(i) of the received image.

In some embodiments, a weighted sum of the top-k vectors is used. Forexample, the weights are given by the similarity (e.g., the similarityscore) of the feature vectors for the images in the database with thefeature vector v_(i) of the received image. For example, if the featurevector of a target image is significantly similar to a feature vector ofan image labeled “selfie” and is not very similar to feature vectors oftwo images labeled “playfield”, then the label “selfie” will be weightedhigher—as a result, the target image can be labeled “selfie” even thoughthat label would be outnumbered using a simple voting scheme of the topk (e.g., 3) results. In some examples, the weighted sum is used inaddition to the simple voting scheme.

In some embodiments, the system stores the determined label. Forexample, the system can store the determined label in a dataset. In someexamples, the label can be associated with the social media post thatincludes the image represented in a dataset.

At box 1112, the system uses the image to build a new classifier. Insome embodiments, building a new classifier with the data from thereceived (and classified) image is optional. For example, if a receivedimage is successfully classified, the data of the image can be added tothe initial dataset (e.g., which would then be composed of 100+1=101images), from which a new classifier can be built (e.g., in accordancewith the process 1000 of FIG. 10.)

The use of images, which have been classified, in order to create a newclassifier (e.g., creating a larger set of data) can provide valuablefeedback to the system. By enhancing the classifier through theintroduction of new data, the accuracy or precision of the classifiercan be increased.

The use of k-Nearest Neighbors classification schemes in order toclassify images according to labels has the advantage that it can beperformed quickly in real time. Another advantage of k-Nearest Neighborsclassification schemes is that they require little training, unlike someother classification schemes. While more complicated classificationschemes may be capable of more accurate or precise results, they do soat the cost of decreased speed and simplicity. Applicants have achievedexcellent results using image classifiers based on k-Nearest Neighborsclassification schemes. The combination of speed, ease of creating andbuilding classifiers, and high quality results makes the use ofk-Nearest Neighbors image classification schemes, according to thetechniques described herein, well-suited for real time handling of largeamounts of social media post data being generated during events.

ii. Deep Learning Image Classifiers

In some embodiments, an image classifier utilizes a deep learningclassification scheme (e.g., an artificial neural network, a deep neuralnetwork, a convolutional neural network). A deep learning classificationscheme can be used to classify an object (e.g., determine a classmembership of an image) using a classification scheme that has beentrained based on an initial set of training samples for which classes(labels) are known.

Application of a classifier that utilizes a deep learning classificationscheme generally involves two processes: “building” the classifier(which can also be referred to as “training” the classifier), andperforming classification using the classifier.

FIG. 12 depicts an exemplary flow diagram of a process 1200 for traininga classifier that utilizes a deep learning classification scheme. Thetraining phase typically involves iteratively applying the deep learningclassification scheme's fundamental computation steps (e.g.,backpropagation) to one or more images in the initial training sampledataset, and adjusting the parameters of the classification scheme(e.g., neural network) until an acceptable error rate is achieved.

At block 1202, a system ingests a set of images. For example, the systemcan ingest the set of images as described above with respect to block902 of FIG. 9. At block 1204, the system pre-processes the set ofimages. For example, the system can pre-process the set of images asdescribed above with respect to block 904 of FIG. 9.

At block 1206, the system processes the set of images using a deeplearning function. For example, if a dataset (e.g., dataset 300 of FIG.3) includes 100 images, these images are processed during an initialiteration of a deep learning function.

At block 1208, the system outputs classification predictions of the deeplearning function. For example, after the initial processing using thedeep learning function, the system outputs a set of class predictionsfor each image in the set of images.

At block 1210, the system determines a classification error rate. Forexample, using data included with the ingested images that indicates thecorrect class labels associated with the images ingested, the systemdetermines the error rate of the deep learning function's output. Anyappropriate technique for determining error can be used. For example,one measure of error is the precision of the classification scheme,which can be defined as:

$E_{precision} = \frac{\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {images}\mspace{14mu} {classified}\mspace{14mu} {correctly}} \right)}{\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {images}\mspace{14mu} {classified}} \right)}$

An alternative measure of error is the recall of the classificationscheme, which can be defined as:

$E_{recall} = \frac{\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {images}\mspace{14mu} {correctly}\mspace{14mu} {assigned}\mspace{14mu} {to}\mspace{14mu} a\mspace{14mu} {class}} \right)}{\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {images}\mspace{14mu} {that}\mspace{14mu} {should}\mspace{14mu} {be}\mspace{14mu} {assigned}\mspace{14mu} {to}\mspace{14mu} {the}\mspace{14mu} {class}} \right)}$

At block 1212, the system determines whether the error rate isacceptable. If yes, then the system proceeds to box 1214, and stores thedeep learning function. If not, the system proceeds to box 1216, andreceives a parameter adjustment. The acceptable error rate can depend onthe particular situation and application, as well as on userpreferences. For example, the system may have a predefined thresholdrepresenting an acceptable error rate.

In some embodiments, the system performs automatic threshold selection.In this example, the classifier would train itself through thresholdselection.

In some embodiments, the system receives a parameter adjustment via userinput. For example, a user can adjust one or more parameters of the deeplearning function (e.g., an artificial neural network).

After block 1216, the system returns to block 1206 and processes the setof images using the deep learning function again. In this example, thesystem received a parameter adjustment to the deep learning function, sothe classification results may be different. As shown in FIG. 12, theprocess 1200 iterates in this manner until an acceptable error rate isachieved.

FIG. 13 depicts an exemplary process 1300 for classifying a target imageusing a classifier that utilizes a deep learning function. At box 1302,a system receives an image. For example, the system receives the targetimage image201.jpeg of dataset 600 (e.g., image 802 of FIG. 8A). In someembodiments, the image (or an address or other identifier of the image)is retrieved via an API of a social media network as described above.The received image may be pre-processed as described above.

At block 1304, the system processes the image using a deep learningfunction. For example, the deep learning function used at block 1304 canbe the deep learning function (having an acceptable error rate) storedat box 1214 of process 1200. In this example, the trained deep learningfunction (e.g., an artificial neural network) is now used to classify animage that is not part of the set of training images (e.g., an imagethat has been pulled from social media post in real time).

At block 1306, the system determines a classification label matchprediction for the image. For example, the result of processing theimage with the deep learning function is used to determine theprediction. In this example, the image201.jpeg of dataset 600 (depictedin image 802 of FIG. 8A) has a class prediction of “selfie” where theset of labels are {“playfield”, “selfie”, “other”}. The prediction canbe a numerical probability that the image matches a given label, or canbe a similarity score between the image and the representative set ofimages of the given label. In some examples, then, each label can have aprediction (e.g., probability and/or similarity score) indicating thelikelihood that the image matches other images from the given class.

At block 1308, the system determines a label for the received image. Insome embodiments, the deep learning function (e.g., artificial neuralnetwork) outputs probabilities for each classification label in the setof labels. In some examples, the highest probability label can beassigned to the image. In some examples, the label with the highestprobability can be assigned to the target image only if the probabilityexceeds a threshold. In some examples, any label can be assigned to animage if its respective probability exceeds a threshold (e.g., the imagecan be classified with two labels). For example, the image 802 of FIG.8A might also have a relatively high probability of matching the“playfield” label, in addition to “selfie”, given that the image depictstwo persons as well as a large amount of the interior of the basketballarena venue (“City Arena”).

In some embodiments, the deep learning function generates a plurality ofoutput results. In some embodiments, a plurality of deep learningfunctions are used to process the received image. In some examples, aclass label is assigned to the received image based on simple voting ofthe plurality of outputs from the one or more deep learningclassification schemes.

At block 1310, the system uses the received image to train the deeplearning function. Using the received image to train the deep learningfunction (e.g., using process 1200) is optional. An increase in thenumber of representative images used to train the classifier, however,can be potentially beneficial for the classification of future images.

iii. “Cold Start” Phase Using Image Classifiers

While k-Nearest Neighbors and deep learning functions are discussedabove, other image classification schemes can be used to create aclassifier for use in accordance with the techniques described herein.In some embodiments, regardless of the underlying technique for imageclassification, a system will perform a “cold start” process. Forexample, if a classifier does not exist, one can be created and tuned inorder to be effective at image classification. Though the cold startphase is broadly applicable to any classifier, it is similar to theprocesses of building the k-Nearest Neighbors classifier or training thedeep learning classifier as described above.

FIG. 14 depicts an exemplary process 1400 for creating a classifier froma set of images using any classification scheme. At block 1402, a systemcreates a dataset from a set of images. For example, the set of imagescan be accessed or retrieved as described above (e.g., via an API, frommemory storage, etc.). Creating the dataset can include the process ofingesting image data, as described above. In some embodiments, creatingthe dataset includes creating a dataset from social media post data. Insome examples, the social media post data includes more than image data(e.g., text, hashtags, geotags, image labels, etc.) The dataset created(e.g., dataset 300, dataset 600) from the set of images is formatted andorganized in any manner that is computer-readable. In some embodiments,the system pre-clusters the images into defined classes.

In some examples, the data ingested to create the dataset is from socialmedia posts associated with images known to have been posted at and/orthat depict a geographic location (e.g., a sporting venue). In someembodiments, the system retrieves posts from one or more social medianetworks, wherein the posts have a geotag (or other locationinformation) identifying a geographic location of interest. For example,the system downloads the images associated with these posts. In someembodiments, the system retrieves the post data from a database of postspreviously retrieved (from social media networks) that are confirmed(e.g., by geotag, or by user confirmation) to have been posted at and/ordepict the geographic location of interest.

At block 1404, the system cleans and normalizes the data. Cleaning andnormalizing the data is meant to ensure that all images of the datasetare represented within common scales of values (e.g., one or more ofsizes, pixel colors, brightness, etc.) and that outliers are explainedand/or removed.

In some embodiments, one or more of the following normalizationtechniques are applied over the image dataset: imposing a common shapefor all images (e.g., 100×100, 320×320), quantizing color data, andimposing a common color space.

In some embodiments, outliers in the dataset are removed or explained.For example, outliers can include: images that are almost entirely blackor entirely white, meaningless images (e.g., blurry, smeared, unclear,thumb on lens, etc.), an image that is repeated multiple times, andimages having nothing to do with the context (this can be subjective andinterpreted differently for different datasets).

In some embodiments, cleaning and normalizing the data is an automatedprocess. In some embodiments, cleaning and normalizing the data relieson user input (e.g., to select irrelevant images as outliers). In someembodiments, some combination of both automation and user input areused.

At block 1406, the system segments the data into classes. In someembodiments, the accessed data (e.g., set of images, social media posts)will already be associated with one or more labels of a set of labels(classes). In some examples, the images associated with known locationswill need to be segmented into classes (also referred to as assigninglabels to the images). Examples of classes include the set of classes{“playfield”, “selfie”, “other”} discussed above. In some embodiments,segmenting the data into classes includes creating a dataset for eachimage class. In some embodiments, segmenting the data into classesincludes creating a dataset that includes segments organized by class.For example, a single dataset can have images of the same class groupedadjacently within the same data structure. In some embodiments, thesystem segments the image data into classes based on user input. Forexample, user input is received that specifies or confirms a classassignment.

Classes can be derived manually (e.g., by iteratively applying dataanalysis and trials), automatically (derived from available data), orboth. Manually deriving a class can include receiving user inputrepresenting a plurality of images for each class label that a userobserves after browsing the training dataset. For example, user inputcan specify 12 images that should be labeled “playfield”. The systemthen creates a classifier (e.g., as described above) based on thesespecified (thus, labeled) images. Automatically creating classes caninclude processing the images to determine groups of related (e.g.,visually similar) images. In some embodiments, deriving classes is acombination between automatic and manual processes. For example,segmenting can be based on some combination of user input (e.g.,verification) and system suggested classes. Segmenting the images intoclasses can be an iterative process—for example, they system can receiveuser input that identifies 5 images that form a class; then a systemsuggests 5 more images, for which user input verification is receivedfor 3 of the images; then the system suggests another 5 images, based onthe 8 verified images; and so on. This process can be repeated untilsatisfactory classes are created and populated.

As described above, for example, the system can use the classifier toidentify additional images in the dataset (or other datasets) to beassociated with each of the classes. The system stops identifying imageswhen a sufficient number of images are found for each class, or when thedataset is exhausted. As can be seen, the process of assigning images toclasses in order to create an initial dataset (used to build aclassifier) is semi-automatic, and is done at initial stages ofclassifier creation, when there is little or no prior knowledge ofimage-label associations. As the knowledge of the system increases,(e.g., through robust classifiers) the process can become moreautomated.

At block 1408, the system applies one or more image classifiers to thedataset until acceptable image classification results are obtained. Forexample, the image classifiers are applied and tuned until an acceptableerror rate is achieved. For instance, the system can modify parametersof the classification scheme(s) until an acceptable level of precisionand/or recall is achieved for a given class. The balance of precisionversus recall can be adjusted to suit different scenarios orpreferences. In some embodiments, the classifier is applied to adifferent dataset (e.g., different than the initial dataset) to test theclassifier's error rate.

In some embodiments, if a k-Nearest-Neighbors classification scheme isused as a classifier, the value of k can be adjusted in order to tunethe classifier parameters. Other exemplary tuning parameters are theselection and application of Latent Semantic Indexing (LSI) functions tobe applied over the contingency values, selection and addition ofvariables based image transformations (e.g., pixel clusters,Red-Green-Blue to Hue-Saturation-Value, etc.), tweaking the significanceof different pixel patches (e.g., the image corners are less importantthat the image center), using matrix factorization techniques forextracting pixel patches topics, using some or all of data obtained froma Scale Invariant Feature Transform (SIFT), or similar functions.

In some embodiments, different classification schemes are used foridentifying each class. For example, one type of classification scheme(e.g., k-Nearest Neighbors) may provide superior results identifyingimages of a first class, whereas another type of classification scheme(e.g., deep learning) may provide superior results identifying images ofa different second class. In some embodiments, multiple classifiers areused together to identify images of the same class. For example, twodifferent classifiers may complement each other to produce superiorresults for a single class, rather than one classifier alone. In thisexample, the results of each classifier can be further processed todetermine the classification decision. For example, a simple votingscheme using the results of each classifier can be used to determine theclassification label.

In some embodiments, the system uses an approximation residual ofdimension reduction singular value decomposition (“SVD”). In someembodiments, the system creates a composite classifier that takes intoaccount weights of its component classifiers.

At block 1410, the system enhances the classifier processes withadditional data. For example, as touched on above, if the classifierclassifies new images from additional data or datasets, those images canbe used to create new, or enhance previously-created, classifiers. Insome embodiments, enhancing the classifier with additional data isoptional.

In some examples, additional data, such as hashtag retrieved messagesthat include images, can be used to further build/train the classifiers.In some embodiments, additional data is evaluated based on whether itwould contribute to enhancing the knowledge of the classifier and avoidthe “saturation point” where: (i) adding new data makes the classifierslower without improving the accuracy (e.g., increases the processingtime required to output a result); or (ii) adding new data over-trainsthe classifier (e.g., produces worse results, such as less accurate).

b. Semantic Analysis

In addition to performing image classification, a system performinglocation resolution in accordance with the techniques described hereincan also apply semantic analysis to the content of a social media post.Semantic analysis (otherwise referred to as “language analysis”) can beused to identify and quantify patterns and similarities in the textualcontent (e.g., text, hashtags) in social media posts. This isparticularly useful, for example, where a social media post does notinclude an image but still contains valuable insight into a customer'suser experience at a geographic location. Relying solely on imageanalysis could result in this content being overlooked, and valuabledata lost. Further, using semantic analysis complements image analysis,such that the results of a geographic location resolution process can beimproved.

Any appropriate semantic analysis technique can be used to analyze thecontent of social media posts. In some embodiments, the system uses acombination of two or more semantic analysis techniques. Two particulartechniques are discussed below, though one skilled in the art wouldrecognize the applicability of other known or future semantic analysistechniques for determining textual similarity between data.

i. Named-Entity Recognition

In some embodiments, semantic analysis includes performing anamed-entity recognition process. Named-entity recognition involves theidentification and classification of named entities in a block of textinto categories such as persons, organizations, locations, expressionsof time, quantities, monetary values, and the like.

FIG. 15 depicts an exemplary process 1500 for performing a named-entityrecognition process on a block of received text. Named-entityrecognition is also referred to as “entity name recognition”. At block1502, the system detects a name. For example, the system detects anentity name in target text. For example, in the social media post number3 of dataset 600 (FIG. 6), the system detects the name “Alex”, whichcorresponds to a person. In another example, the system detects the name“Team” in social media post number 8 of dataset 600, which correspondsto an organization.

At block 1504, the system classifies the name. For example, the systemclassifies the name “Alex” as a person. In another example, the systemclassifies the name “Team” as an organization. In some embodiments,detection (e.g., block 1502) and classification (e.g., block 1504) of aname are performed together and/or are part of the same process. Forexample, the name Alex can be predefined as being in the class“persons”. In other words, the detection of the Alex as a name caninherently entail the identification that the name is a person.

In some embodiments, the system performs the techniques described abovewith reference to blocks 1502 to 1504 for a plurality of named entitiesin the block of received text. In some embodiments, the block ofreceived text includes data from one or more social media posts.

At block 1506, the system determines a similarity measure between thedetected name and a representative set of names. In some embodiments,the system determines a similarity measure between a plurality ofdetected names and the representative set of names. For example, thesystem can compare the name “Alex” or “Team” to a representative set ofnames. The representative set of names is, for example, datarepresenting one or more social media posts of a known location. The oneor more social media posts should be posts of a known location, suchthat the system compares a post of an unknown location to those of knownlocation. For instance, by analyzing the post text “Team wins!!”, thesystem determines the similarity between the name “Team” and a set ofposts of known location, such as the posts of dataset 300. In thisexample, post 1 in dataset 300 includes the text, “What a beautiful newarena! Can't wait to watch Team play tonight!”. Thus, because the phrase“Team” appears in both texts, the system determines a similarity measurereflecting that these posts are similar.

The measure of similarity between the detected name and a representativeset of names can be expressed in one or more appropriate ways. In someembodiments, the similarity is expressed as a single value. For example,the value can be the total number of occurrences of the name in the setof names (e.g., names within posts of known location). In otherexamples, the value is a score based on the occurrence of the name inthe set of names. For example, the score can be based on the proportionof social media posts of a known location that include the name. Inother examples, the score is based on the occurrence of the name withinthe representative set of names, relative to a second set ofrepresentative names (e.g., social media posts of a different knownlocation). One of skill in the art would readily appreciate that thereare many other ways to express a similarity measure, any of which areintended to be within the score of this disclosure.

In some embodiments, the similarity measure value can be a plurality ofvalues. For example, the values can each correspond to the total numberof occurrences of each of a plurality of detected names in the set ofnames.

In some embodiments, the similarity measure is a combined metric basedon the occurrence of the one or more detected names in the set ofrepresentative names. For example, if multiple detected names from apost being analyzed match the set of names, a combined metric could behigher than if only one detected name matches a name in the set ofrepresentative names.

At block 1508, the system outputs a result. In some embodiments, thesystem outputs the similarity measure. The output (e.g., similaritymeasure) can be used, in accordance with other data, in thedetermination of a location for the social media post being analyzed.

ii. Topic Model/Topic Extraction

In some embodiments, semantic analysis includes performing a topic modelextraction process. A topic model is a statistical model that can beused to determine abstract “topics” that occur in blocks of text. Forexample, in blocks of text about the topic of a basketball game, thefollowing words would be expected to occur together: “basketball”,“court”, “tip off”, “bucket”, “dunk”, “shot”, and other words associatedwith the game of basketball. By identifying these word clusters in agiven block of text, a determination can be made of the likelihood thata given social media post (containing those words) relates to aparticular topic (e.g., basketball). In addition to the presence oftopic-related words, the semantic structure of the words can also beused as part of the topic model analysis. If the topic is indicative ofan event or a particular location, then this determination can aid inthe location resolution process.

FIG. 16 depicts an exemplary process 1600 for a topic extraction processon a block of received text in accordance with some embodiments. Atblock 1602, the system determines a topic model. For example, to analyzewhether a given social media post relates to a basketball game, thesystem determines the topic model “basketball”. Determining the topicmodel can include, for example, accessing data representing that topicmodel. In some embodiments, the topic model data is stored as a datastructure and retrieved from memory. The topic model data can be storedremotely or locally. In some embodiments, determining the topic modelincludes building the topic model from a dataset of text. For example,the text of dataset 300 can be analyzed in order to extract a topicmodel for the topic “basketball”. In some embodiments, a plurality oftopic models is determined.

At block 1604, the system analyzes target text to determine a similaritymeasure with the topic model. For example, the system analyzes thesocial media post text and hashtags from post 2 of dataset 600 (FIG. 6)to determine their similarity to the topic “basketball”. In someembodiments, the system extracts words from the target text of the post.For example, the system can extract the words “game”, “Team”, and“basketball” from the text and hashtags of the post. Using this group ofwords, or some subset or superset thereof, the system determines asimilarity measure with the topic model (e.g., derived from an initialdataset of the social media posts from the known location). In someembodiments, the system analyzes the target text to determine aplurality of similarity measures with a plurality of topic models,respectively.

The measure of similarity between the target text and the topic modelcan be expressed in one or more appropriate ways. In some embodiments,the similarity is expressed as a single value. For example, the valuecan be the total number of occurrences of words in the target text thatoccur in the topic model. In other examples, the value is a score basedon the occurrence of words shared with the topic model. For example, thescore can be based on the proportion of words in the target text thatmatch the topic model relative to the number of words that do not matchthe topic model. In some examples, the similarity score is non-linearwith respect to the topic overlap between the topic model and thetext—that is, the similarity can rise multiplicatively as the proportionof target text that matches the topic model increases. One of skill inthe art would readily appreciate that there are many other ways toexpress a similarity measure, any of which are intended to be within thescope of this disclosure.

At block 1606, the system outputs a result. In some embodiments, thesystem outputs the similarity measure. The output (e.g., similaritymeasure) can be used, in accordance with other data, in thedetermination of a location for the social media post being analyzed.

3. Location Resolution

Up to this point, various specific concepts related to the retrieval,storage, and analysis of social media post content have been described.These concepts, used alone, might reveal limited insights intorelationships between social media post data. However, determining thegeographic location associated with social media posts that lack anassociated geographical reference in a reliable and useful way (e.g., inreal time) presents a particularly difficult challenge. An exemplarylocation resolution technique is described below.

FIG. 17 depicts an exemplary process 1700 for location resolution of asocial media post, in accordance with some embodiments. At block 1702,the system accesses data representing a first social media post. In someembodiments, the system accesses data representing a first social mediapost that is associated with a geotag, or which has otherwise beenverified to have been posted at and/or depict a geographic location. Forexample, the system accesses social media post number 2 of dataset 300(FIG. 3), which is also post number 4 of dataset 600 (FIG. 6). Asdescribed above, accessing the post can include retrieving datarepresenting the post from storage (e.g., stored as a dataset), from asocial media network (e.g., via an API), or the like. In someembodiments, the data representing the first post includes an imageand/or a reference to an image. In some embodiments, the datarepresenting the first post includes a video and/or a reference to avideo.

At block 1704, the system identifies a second social media post relatedto the first social media post. For example, the second social mediapost can be related the first social media post by virtue of having beenposted by the same user account, and near in time to the first post.Further, in this example, the second social media post does not includean associated geographic location in its representative data (e.g., thepost does not include a geotag). For instance, the system identifies anyof posts 1-3 and/or 5-8 of dataset 600 as a related post (e.g., postedby the same user). For the sake of this example, the system identifiespost 3 of dataset 600, which includes the text, “At the game with AlexB.” Specific exemplary techniques for identifying related social mediaposts are discussed below, with reference to FIG. 18.

At block 1706, the system access data representing the second socialmedia post. As described above, accessing post data can includeretrieving data representing the post from storage (e.g., stored as adataset), from a social media network (e.g., via an API), or the like.For example, accessing the data can include retrieving an imageassociated with the data, where the data contains a reference or link tothe image.

At block 1708, the system analyzes the data representing the secondsocial media post. In some embodiments, analyzing the data representingthe second social media post includes performing an image classificationprocess to an image associated with the post (e.g., included in the datarepresenting the second post). For example, the system can perform oneor more of exemplary processes 1100 (FIG. 11) and 1300 (FIG. 13). Inthis example, the image associated with the identified second post thatis analyzed is image 802 of FIG. 8A, which depicts two people posingwhile at a basketball game at City Arena.

In some embodiments, analyzing the data representing the second socialmedia post includes performing a semantic analysis process to textassociated with the post (e.g., included in the data representing thesecond post). For example, the system can perform one or more ofexemplary processes 1500 (FIG. 15) and 1600 (FIG. 16). In this example,the text associated with the identified second post that is analyzed is“At the game with Alex B.” In some examples, the system also performssemantic analysis on hashtags associated with a post.

At block 1710, the system determines a location score. In someembodiments, the location score is determined based on analysis of thedata representing the second social media post. For example, thelocation score can be based on the output of one or more of the imageclassification and the semantic analysis processes. In some embodiments,other values are taken into account to determine the location score.Exemplary other values include a measure representing the separation intime between when the second social media post was posted and when the(related) first social media post of known location was posted. Putanother way, if the posts are 5 hours apart and an event of interest atthe geographic location is only 1 hour in length, there is a smallerlikelihood that the non-geotagged post was posted from the geographiclocation, which can be reflected in the location score. Thus, in someexamples, the location score can be thought of as a confidence scorethat the second social media post was also posted at and/or depicts thegeographic location with which the related first post is associated.

At block 1712, the system determines whether the location score exceedsa threshold score. For example, the threshold score can be a predefinedvalue.

At block 1714, if the location score exceeds the threshold score, thesystem associates the second social media post with the first geographiclocation. For example, if the combination of the image classification,semantic analysis, and other relevant factors combine for a locationscore that exceeds the threshold, the second social media post isassigned the same geographic location as the first social media post.Thus, if (1) the image analysis of image 802 of FIG. 8A (e.g., from thesecond post) is determined to likely depict the interior of City Arena,and (2) the words and hashtags used in the second post match other postsknown to have been posted at the basketball arena, then (3) the locationscore will likely exceed the threshold and the second post will beassociated with the geographic location “City Arena”.

At block 1716, if the location score does not exceed the thresholdscore, the system forgoes associating the second social media post withthe first geographic location.

In some embodiments, a first user account posted the first social mediapost, and identifying the second social media post related to the firstsocial media post comprises: determining a window of time based on atime associated with the first social media post; and identifying one ormore social media posts posted by the first user account during thewindow of time, wherein the one or more social media posts includes thesecond social media post. For example, if post 4 of dataset 600 is thefirst post, the system determines a window of two hours in length (onehour before and one hour after the time of post 4, Aug. 13, 2018 at 7:30PM). Post 3 of dataset 600 would be identified as the second postbecause it falls within the two hour window (it was posted only fifteenminutes before the first post, at 7:15 PM on the same day).

In some embodiments, a first user account posted the first social mediapost, and identifying the second social media post related to the firstsocial media post comprises: determining that a second user account ismentioned and/or tagged in the first social media post; and identifyingone or more social media posts posted by the second user account,wherein the second social media post was posted by the second useraccount, and wherein the first and second user accounts are different.For example, if post 3 of dataset 600 were taken to be the first socialmedia post (assuming it had an associated geographic location), then thesystem would identify the second social media post from posts by AlexB., who is mentioned in the text of the first post, post 3.

In some embodiments, identifying the second social media post related tothe first social media post comprises: determining that the first socialmedia post includes a first tag; and identifying the second social mediapost based on the first tag, wherein the second social media postincludes the first tag. For example, post 2 of dataset 600 can beidentified as a related social media post to post 4 of dataset 600because it includes the shared hashtag “# GoTeam”.

In some embodiments, a first user account posted the first social mediapost, and identifying the second social media post related to the firstsocial media post comprises: determining that a third user accountinteracted with the first social media post; and identifying one or moresocial media posts posted by the third user account, wherein the secondsocial media post was posted by the third user account, and wherein thefirst and third user accounts are different. For example, the systemidentifies the second social media post from the posts of a user accountwho liked, shared, commented on, or otherwise interacted with the firstsocial media post (e.g., posted by a different user account).

In some embodiments, identifying the second social media post related tothe first social media post comprises: accessing a database ofidentifiers associated with users previously identified as likely to belocated at the first geographic location; matching a fourth user accountto an identifier in the database; and identifying one or more socialmedia posts posted by the fourth user account, wherein the second socialmedia post was posted by the fourth user account. For example, thesystem determines that the email address associated with the useraccount matches a database of identifiers (e.g., email addresses) ofpersons who purchased tickets to an event (e.g., that will be held atthe geographic location).

In some embodiments, analyzing the data representing the second socialmedia post comprises: performing, by one or more processors, semanticanalysis to determine a semantic similarity score between the secondsocial media post and a collection of data representing social mediaposts identified as being associated with the first geographic location;wherein the location score is determined based at least in part on thesemantic similarity score.

In some embodiments, if the location score for the data representing thesecond social media post exceeds the threshold location score, thesystem adds the data representing the second social media post to thecollection of data representing social media posts identified as beingassociated with the first geographic location. For example, the systemadds data representing the second social media post to a dataset ofposts of known location.

In some embodiments, the data representing the second social media postincludes an image, and analyzing the data representing the second socialmedia post comprises: applying, by one or more processors, a computervision classification algorithm to the image of the data representingthe second social media post to determine a first class confidencescore; if the first class confidence score for the image of the datarepresenting the second social media post exceeds a threshold firstclass confidence score, classifying the image of the second social mediapost as matching the first class, wherein the location score isdetermined based at least in part on the classification of the image ofthe second social media post as matching the first class; if the firstclass confidence score for the image of the data representing the secondsocial media post does not exceed the threshold first class confidencescore, forgoing classifying the image of the second social media post asmatching the first class. For example, the system applies one or moreimage classifiers (e.g., such as those discussed above) to an image ofthe second social media post.

In some embodiments, the computer vision classification algorithmincludes one or more of: a nearest-neighbor classification algorithm andan artificial neural network classification algorithm. For example, thesystem applies one or more of a k-Nearest Neighbor or deep learningbased classifier, as discussed above.

In some embodiments, if the location score for the data representing thesecond social media post exceeds the threshold location score, thesystem uses the data representing the second social media post to updatethe computer vision classification algorithm. For example, the systemenhances one or more classifiers using the data of the second socialmedia post.

In some embodiments, the system accesses, by one or more processors,data representing a third social media post, wherein the datarepresenting the third social media post does not include geographiclocation data identifying the first geographic location, and wherein thedata representing the third social media post includes an image; andapplies, by one or more processors, the updated computer visionclassification algorithm to the image of the data representing the thirdsocial media post. For example, the system performs location resolutionon an additional, third social media post using the classifier enhancedwith the data from the second social media post.

In some embodiments, the computer vision classification algorithm relieson at least one image that matches the first class and that is notassociated with the first geographic location. For example, the imageclassifier is built from (or trained using) an image from (or depicting)a geographic location other than the geographic location associated withthe related post, but that matches a common class label. That is, animage from a different venue, “National Arena”, is used to classify theimage of the second social media post, which depicts “City Arena”. Theimages, however, both belong to the class “playfield”.

In some embodiments, if the location score for the data representing thesecond social media post does not exceed the threshold location score,the system forgoes associating the second social media post with thefirst geographic location. For example, if the results of the analysisdo not tend to show that the social media post was posted from thegeographic location, then the system does not associate location data ofthe first geographic location with the post (e.g., in system memory, ina dataset, etc.).

FIG. 18 depicts exemplary processes for determining a second socialmedia post related to a first social media post. One or more ofprocesses 1810, 1820, 1830, 1840, and 1850 can be used at block 1704 ofprocess 1700 (FIG. 17).

Blocks 1812 to 1814 of process 1810 depict a technique that utilizes awindow of time when determining a related social media post. At block1812, the system determines a window of time. In some embodiments, thewindow of time corresponds to the length of an event at the geographiclocation. For example, if an event at a geographic location is scheduledto last for a 4 hour window from 5:00 PM to 9:00 PM, then the windowduring which social media posts related to the event may be 6 hours(e.g., 4:00 PM to 10:00 PM).

At block 1814, the system identifies the second social media post from aset of posts from the window of time. For example, if the first socialmedia post is post 4 of dataset 600 of FIG. 6, which was posted at 7:30PM on Aug. 13, 2018, then the social media post 3 would satisfy thewindow of time criteria given above because it was posted at 7:15 PM onAug. 13, 2018.

Blocks 1822 to 1824 of process 1820 depict a technique that utilizesuser accounts referenced in the first social media post. At block 1822,the system determines a second user account mentioned and/or tagged inthe first social media post, wherein the first social media post wasposted by a first user account. For example, if post 3 of dataset 600was posted by user “Cindy J.” (e.g., the first user account), the systemdetermines that the user “Alex B.” was tagged and mentioned in the post(e.g., and thus Alex B. is the second user account).

At block 1824, the system identifies the second social media post fromposts posted by the second user account. For example, the second socialmedia post would be identified by posts made by the user account Alex B.For instance, the system identifies posts by Alex B. that were postedaround the same time as the first social media post (e.g., within awindow of time).

Blocks 1832 to 1834 of process 1830 depict a technique that utilizeshashtags to identify a second social media post. At block 1832, thesystem determines a first tag. For example, the system determines thatthe post 4 of dataset 600 includes the hashtags “# Gametime” and “#GoTeam”.

At block 1834, the system identifies the second social media post basedon the first tag. For example, the system identifies post 6 of dataset600 as the second social media post, which includes the common hashtag“# GoTeam”.

Blocks 1842 to 1844 of process 1840 depict a technique that utilizesuser accounts that interacted with a first social media post. At block1842, the system determines a user account that interacted with thefirst social media post. For example, if a user account (e.g., Alex B.)likes, shares, comments, or otherwise interacts publically with thefirst social media post (e.g., posted by Cindy J.), the systemdetermines that Alex B. interacted with the first social media post.

At block 1844, the system identifies the second social media post fromposts posted by the identified user account. In this example, the systemidentifies a post that was made by Alex B. In some examples, theidentified post was posted around the same time as the first socialmedia post (e.g., within a window of time).

Blocks 1852 to 1856 of process 1850 depict a technique that utilizes adatabase of identifiers associated with known potential attendees at ageographic location of interest. At block 1852, the system accesses adatabase of identifiers associated with potential attendees of the firstgeographic location. In some embodiments, potential attendees arepurchasers of a ticket to an event at the geographic location. In someembodiments the identifiers are email addresses, user names, accountnames, or other identifiable monikers that are provided during a ticketpurchase transaction. For example, if a customer purchases a ticket froman online ticket seller, the user may have created an account (e.g.,with a user name) or provided an email address (e.g., at which toreceive the tickets). If the user also maintains a social media account(e.g., user account) that shares the email address, and the social mediaposts shared are accessible (e.g., publically-viewable), then the systemdetermines that the user (of the user account) is likely to physicallybe in attendance at the geographic location at the time of the event.

At block 1854, the system matches a user account to an identifier in thedatabase. As described above, the system can determine if potentialattendees (e.g., ticket purchasers) have a social media user accountthat is publically-viewable and that matches information provided duringthe ticket purchase.

At block 1856, the system identifies the second social media post fromposts posted by the identified user account. For example, if the useraccount (Cindy J.) associated with post 3 of database 600 had purchasedthe ticket to the event using an email address common to her socialmedia account, and the social media account was publically-viewable,then the related post is identified based on this information.

FIG. 19 depicts an exemplary process 1900 for determining a locationscore of a target social media post in accordance with some embodiments.At block 1902, the system receives an image analysis score. For example,the system analyzes image 802 of FIG. 8A and may produce an imageanalysis score of between 0 (e.g., no similarity to a database of knownimages) and 1 (e.g., a perfect match to one or more images in thedatabase of known images). For the sake of example, the image 802produces an example image analysis score of 0.4, around the middle ofthis range, when image classification is performed based on the datasetof images in FIG. 5. Image 802 includes some features that match theimages depicted in FIG. 5, namely structural elements of City Arena(e.g., the upper-level seats) as well as fans in the seats (e.g.,dressed in team colors). However, image 802 does not include a directview of the basketball court and/or other identifiable team logo, so itsimage analysis score falls somewhere in the middle of the 0 to 1 range.In some examples, if the target image includes a clear shot of thebasketball court and team logo (e.g., image 806 of FIG. 8B), then theimage analysis score would be relatively higher (e.g., 0.8).

In some embodiments, if a target social media post (e.g., a social mediapost of unknown location) does not include an image, an image analysisscore is not taken into account (e.g., can be set to zero or assigned noweight).

In some embodiments, the image analysis score corresponds to the outputof an image classifier. In some embodiments, the image analysis scorecorresponds to the output of a plurality of image classifiers. In someexamples, the image analysis score is a value (or sum of values) thatare output from one or more image classifiers. In some examples, theimage analysis score is a function of the outputs from one or more imageclassifiers (e.g., the output is scaled; simple voting scheme).

At block 1904, the system receives a semantic analysis score. Forexample, the text of the social media post (post 3 of dataset 600)corresponding to image 802 of FIG. 8A reads “At the game with Alex B.”The mention of the word “game” and/or the phrase “at the game” willgenerate a high semantic analysis score (e.g., 0.7) when analyzed forsimilarity to the text of the posts in dataset 300.

In some embodiments, the semantic analysis score corresponds to theoutput of a semantic analysis process. In some embodiments, the semanticanalysis score corresponds to the output of a plurality of semanticanalysis processes.

At block 1906, the system receives other relevant data. For example,other relevant data can include any data that the system is programmedto take into account that will affect the location score. For instance,other relevant data can include one or more of: time (e.g., during anevent or other window of time that includes a known geotagged post),geo-spatial temporal statistics, social graph connections (e.g.,associations between users based on available datasets analysis ormessage on social media networks), and other relevant data.

As an example of using geo-spatial temporal statistics as relevant data,a geographic location (e.g., a city) is divided into square regions, anda period of time (e.g., 24 hours) is divided into intervals of time.This will form a space-time coordinate system comprised of space-time“cubes”. Each user account can have a trajectory of “cubes” for theirdaily activity (e.g., as they post around the city over the course ofthe day). Similarly, an event can have a trajectory (e.g., a straightline if the event is at the same location, or a series of adjacent“cubes” for an event like a parade that moves around the city). Userswho are (or have been) at a given event (at a geographic location) havespace-time trajectories that intersect with the event's trajectory.These trajectories (e.g., user, event) and their intersections can beutilized, for example, as part of a location prediction system.

As an example of using social graph connections, and referring back topost 3 of dataset 600, if the user account Alex B. that ismentioned/tagged in the text of the post also checked into the game witha (geotagged) social media post, then this social graph connectioninformation increases the likelihood that the user (Cindy J.) thatposted target social media is physically at the event venue (e.g.,because Alex B. is known to be at the game).

In some embodiments, the system assigns the other relevant data a score.For example, the social graph connection above between Alex B. and CindyJ. could be assigned as score of 0.1, which reflects the increase inlikelihood that Cindy J. is posting from the basketball game.

In some embodiments, the system uses the other relevant data to adjustone or more of the image analysis score, semantic analysis score, or acombination of the image analysis and semantic analysis scores. Forexample, the other relevant data can be used to determine a scalingfactor, which is multiplied by the analysis scores. For instance, if thescaling factor is 1.05, then a location score of 1.1 becomes 1.155.

At block 1908, the system determines a location score for the socialmedia post. For example, the system processes one or more of the imageanalysis score, the semantic analysis score, and the other relevant datainto a location score. In some examples, each score is simply added up.In this example, addition of the three scores would be(0.4+0.7+0.1=1.2).

In some embodiments, the location score is a weighted combination of itscomponent scores. For example, the image analysis can be given a higherweight so that it affects the location score more strongly. Forinstance, using the above example, the location score is now given ahigher weighting (0.5) as compared to the two other factors (eachweighted at 0.25): ((0.4)*(0.5)+(0.7)*(0.25)+(0.1)*(0.25)=0.4). As canbe seen, the heavier weighting of the image score causes the locationscore to be lowered to 0.4.

The location score can be used to assign a geographic location to atarget social media post (e.g., as described above with reference toprocess 1700 of FIG. 17).

4. Using Data Representing Social Media Posts Having a Resolved Location

Determining a physical, geographic location of a social media post isvery valuable, for example, to a business that has a customer-facingphysical presence (e.g., a geographic location such as a venue,storefront, event space, etc.). The geographic location information canbe further utilized to generate new and useful data. Thus, by using asmall pool of data (e.g., geotagged posts) to derive a larger pool ofdata, more content is available to the system for performing furtheranalysis, engaging with more users, and the like.

Described below are several exemplary techniques for using geographiclocation data associated with social media posts to cause a system totake action, to derive additional useful data, or both.

a. Engaging with User Accounts

Using location resolution to determine geographic location data forsocial media posts that would otherwise be overlooked provides theability to engage with a broader base of users and generate more usefuldata regarding user engagement.

FIG. 20 depicts an exemplary process 2000 for determining a Return onEngagement. A Return on Engagement (“ROE”) is a measure of users whoreact with positive actions after being proactively engaged on socialmedia, and thus the effectiveness of social media campaigns.

At block 2002, the system determines a location of one or morenon-geotagged social media posts. For example, the system can performone or more of the location resolution processes described above (e.g.,processes 1700, 1800) to determine a geographic location for one or moresocial media posts that do not include explicit associated locationdata. In some embodiments, the system accesses one or more social mediaposts (e.g., data, or a dataset, representing the posts) whose locationshave previously been determined.

At block 2004, the system engages one or more user accounts of the postsusing a social media network. For example, the system (e.g.,automatically and proactively) interacts with the social media useraccount whose social media post was identified as corresponding to ageographic location. In some embodiments, engaging can include one ormore of: creating a post on the user account's page (e.g., on their“timeline”, “wall”, or the like), interacting with a post created by theuser account (e.g., by marking the post with a “like”, “favorite”,“heart”, “retweet”, or the like), commenting on a post by the useraccount, sending the user account a private message, a combination ofone or more of these actions, or other actions available on social medianetworks that allow interaction between user accounts.

At block 2006, the system analyzes the social media engagements. In someembodiments, analyzing the social media engagements includes monitoringthe engagement for a user response or action.

At block 2008, the system determines the number of users who reactpositively to the engagement. For example, positive reactions include:the user account posting a comment mentioning the social media useraccount that engaged with them (e.g., the venue's social media profile),the user “following” the social media user account that engaged withthem, the user account creating a post again mentioning the social mediaaccount that engaged with them, or the like.

At block 2010, the system determines a Return on Engagement. In someembodiments, the ROE is:

${ROE} = \frac{\begin{matrix}\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {user}\mspace{14mu} {accounts}\mspace{14mu} {who}}\mspace{11mu} \right. \\\left. {{react}\mspace{14mu} {to}\mspace{14mu} {an}\mspace{14mu} {engagement}\mspace{14mu} {with}\mspace{14mu} {positive}\mspace{14mu} {action}} \right)\end{matrix}\;}{\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {user}\mspace{14mu} {accounts}\mspace{14mu} {engaged}\mspace{14mu} {with}} \right)}$

b. Determining a Social Activity Index for a Geographic Location

Using location resolution to determine geographic location associatedwith social media posts that would otherwise be overlooked by othertechniques provides the ability to derive and create actionable datathat otherwise would not be available.

FIG. 21 depicts an exemplary process 2100 for determining a SocialActivity Index for a geographic location. A Social Activity Index(“SAI”) is a measure of the influence that a geographic locationexhibits on social media, relative to its size with regard to similargeographic locations. The SAI standardizes the measurement of socialactivity from events and venues.

At block 2102, the system determines a location of one or morenon-geotagged social media posts. For example, the system can performone or more of the processes described above (e.g., processes 1700,1800) to determine an assigned geographic location for one or moresocial media posts that do not include explicit associated locationdata. In some embodiments, the system accesses one or more social mediaposts (e.g., data, or a dataset, representing the posts) whose locationshave previously been determined.

At block 2104, the system accesses information regarding a geographiclocation. In some embodiments, the information regarding the geographiclocation includes a measure of the physical capacity of the geographiclocation. For example, if the geographic location is a hotel, thecapacity measure can be the total number of hotel rooms. Likewise, ifthe geographic location is a venue such as a stadium or an arena, thecapacity measure can be the total number of seats in the venue. In someembodiments, the information regarding the geographic location includesa measure of the amount of social media activity generated from (orrelated to) the geographic location. For example, this can be the totalnumber of social media posts associated the geographic location. Forinstance, the total number of posts can be the sum of the number ofgeotagged posts and the number of non-geotagged posts whose location hasbeen resolved.

At block 2106, the system accesses information regarding a set ofgeographic locations. In some embodiments, the information regarding theset of geographic locations includes a measure of the physical capacityof the set of geographic locations. For example, if the set ofgeographic locations are hotels, the capacity measure can be the totalnumber of hotel rooms for all of the hotels that are represented withinthe set of geographic locations. The set of geographic locations caninclude the geographic location (e.g., from block 2104). In some otherexamples, if the set of geographic locations are venues such as stadiumsor arenas, the capacity measure can be the total number of seats in thevenues. In some embodiments, the information regarding the set ofgeographic locations includes a measure of the amount of social mediaactivity generated from (or related to) the set of geographic locations.For example, this can be the total number of social media postsassociated the geographic locations included in the set of geographiclocations. For instance, the total number of posts can be the sum of thenumber of geotagged posts and the number of non-geotagged posts whoselocation has been resolved.

At block 2108, the system determines an expected social media influenceof the geographic location. For example, for the geographic locationwhose information was accessed at block 2104, the system determines thesocial media influence expected for the geographic location, based oncapacity. For instance, an expected social media influence for thegeographic location can expressed as:

$I_{expected} = \frac{\left( {{Capcity}\mspace{14mu} {measure}\mspace{14mu} {of}\mspace{14mu} {the}\mspace{14mu} {geographic}\mspace{14mu} {location}} \right)}{\left( {{Total}\mspace{14mu} {capcity}\mspace{14mu} {measure}\mspace{14mu} {of}\mspace{14mu} {the}\mspace{14mu} {set}\mspace{14mu} {of}\mspace{14mu} {geographic}\mspace{14mu} {locations}} \right)}$

From the expression above, a geographic location is expected to producean amount of social media influence relative to a collection ofgeographic locations (e.g., similar venues) in proportion to its size.For example, if a venue has 10,000 seats, and all similar venues containa total number of seats 100,000, then the venue would be expected toproduce 10% of the social media activity related to the set of venues(I_(expected)=0.1). Exemplary expressions are given below for a stadium,and for a hotel. One skilled in the art would recognize that expectedinfluences can similarly be denoted for other types of geographiclocations (e.g., venues, events) based on capacity or size.

$I_{{expected},{stadium}} = \frac{\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {seats}\mspace{14mu} {in}\mspace{14mu} {stadium}} \right)}{\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {seats}\mspace{14mu} {in}\mspace{14mu} a\mspace{14mu} {set}\mspace{14mu} {of}\mspace{14mu} {stadiums}} \right)}$$I_{{expected},{hotel}} = \frac{\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {rooms}\mspace{14mu} {in}\mspace{14mu} {hotel}} \right)}{\left( {{Total}\mspace{14mu} \# \mspace{14mu} {of}\mspace{14mu} {rooms}\mspace{14mu} {in}\mspace{14mu} a\mspace{14mu} {set}\mspace{14mu} {of}\mspace{14mu} {hotels}} \right)}$

At block 2110, the system determines an aggregate number of social mediaposts identified as at the geographic location. For example, the systemdetermines the total number of social media posts (e.g., posts with ageotag and/or a resolved location) posted from (or whose posts relateto) the geographic location (e.g., venue) whose information was accessedat block 2104. In some embodiments, the system determines the totalnumber of social media posts over a predefined period of time. Forexample, the system may determine the total number of posts in the past24 hours, 1 week, 1 month, or the like.

At block 2112, the system determines an aggregate number of social mediaposts identified as at the set of geographic locations. For example, thesystem determines the total number of social media posts (e.g., postswith a geotag and/or a resolved location) identified as having beenposted from (or related to) any of the set of geographic locations. Inthe venue example, all social media posts from all of the venues in theset are aggregated into a total. In some embodiments, the systemdetermines the total number of social media posts over a predefinedperiod of time. For example, the system may determine the total numberof posts in the past 24 hours, 1 week, 1 month, or the like.

At block 2114, the system calculates a social activity index of thegeographic location. In accordance with some embodiments, the SAI can becalculated as follows:

${{SAI} = \frac{\left( I_{activity} \right)}{\left( I_{expected} \right)}},{wherein},{I_{activity} = \frac{\begin{matrix}\left( {{Aggregate}\mspace{14mu} {number}\mspace{14mu} {of}\mspace{14mu} {social}\mspace{14mu} {media}\mspace{14mu} {posts}} \right. \\\left. {{identified}\mspace{14mu} {as}\mspace{14mu} {at}\mspace{14mu} {the}\mspace{14mu} {geographic}\mspace{14mu} {location}} \right)\end{matrix}}{\begin{matrix}\left( {{Aggregate}\mspace{14mu} {number}\mspace{14mu} {of}\mspace{14mu} {social}\mspace{14mu} {media}\mspace{14mu} {posts}} \right. \\\left. {{identified}\mspace{14mu} {as}\mspace{14mu} {at}\mspace{14mu} {the}\mspace{14mu} {set}\mspace{14mu} {of}\mspace{14mu} {geographic}\mspace{14mu} {locations}} \right)\end{matrix}}}$

For instance, in the example above, the stadium has an I_(expected)=0.1.Thus, it is expected that the proportion of social media activitygenerated from the stadium will be 10% of all activity for a set ofstadiums (i.e., that I_(activity) will also be equal to 0.1). However,if the aggregate number of posts identified as at (or related to) thestadium is 20% of all the aggregate number of posts identified as at (orrelated to) a set of stadiums (that includes the stadium), thenI_(activity)=0.2. Thus, according to the expression above, the stadiumwould have an SAI=(0.2)/(0.1)=2. In some examples, an SAI of greaterthan 1 indicates a higher than expected amount of social activity thatoriginates from or is related to the geographic location.

c. Determining a Social Influence Index for a Geographic Location

Using location resolution to determine geographic location associatedwith social media posts that would otherwise be overlooked by othertechniques provides the ability to derive and create actionable datathat otherwise would not be available.

FIG. 22 depicts an exemplary process 2200 for determining a SocialInfluence Index for a geographic location. A Social Influence Index(“SIT”) is a measure of the influence that users posting from (or about)a geographic location exhibit on social media, relative to its sizeand/or similar geographic locations. This is related to the social“reach” that a geographic location has—e.g., the number of users thatcan be reached through social media engagement within one degree ofseparation from the geographic location on one or more social medianetworks. The SII standardizes the measurement of social influence(potential reach) of events and venues.

At block 2202, the system determines a location of one or morenon-geotagged social media posts. For example, the system can performone or more of the processes described above (e.g., processes 1700,1800) to determine an assigned geographic location for one or moresocial media posts that do not include explicit associated locationdata. In some embodiments, the system accesses one or more social mediaposts (e.g., data, or a dataset, representing the posts) whose locationshave previously been determined.

At block 2204, the system accesses information regarding a geographiclocation. In some embodiments, the information regarding the geographiclocation includes a measure of the physical capacity of the geographiclocation.

At block 2206, the system identifies one or more user accounts on asocial media network that have posted from the geographic location. Forexample, the system determines all user accounts that have createdpublic social media posts at (e.g., geotagged) or related to a venue. Insome embodiments, the system identifies user accounts from a pluralityof social media networks.

At block 2208, the system determines an aggregate number of useraccounts that are associated with the user accounts that have postedfrom (or about) the geographic location. For example, the systemdetermines an aggregate number of one or more of the following:followers, friends, subscribers, or the like, of the user accounts thathave posted from the geographic location.

At block 2210, the system identifies one or more user accounts on asocial media network that have posted from a set of geographiclocations. For example, the system determines all user accounts thathave created public social media posts at (e.g., geotagged) a set ofsimilar venues. In some embodiments, the system identifies user accountsfrom a plurality of social media networks.

At block 2212, the system determines an aggregate number of useraccounts that are associated with the user accounts that have postedfrom the set of geographic locations. For example, the system determinesan aggregate number of one or more of the following: followers, friends,subscribers, or the like, of the user accounts that have posted from theset of geographic locations.

At block 2214, the system calculates a Social Influence Index of thegeographic location. In accordance with some embodiments, the SocialInfluence Index (“SII”) can be calculated as follows:

$\mspace{79mu} {{{SII} = \frac{\left( I_{influence} \right)}{\left( I_{expected} \right)}},{wherein},{I_{influence} = \frac{\begin{pmatrix}{{{Aggregate}\mspace{14mu} {number}\mspace{14mu} {of}\mspace{14mu} {user}\mspace{14mu} {accounts}\mspace{14mu} {assoicated}\mspace{14mu} {{wit}h}\mspace{14mu} {the}}\mspace{14mu}} \\{{user}\mspace{14mu} {accounts}\mspace{14mu} {that}\mspace{14mu} {have}\mspace{14mu} {posted}\mspace{14mu} {from}\mspace{14mu} {the}\mspace{14mu} {geographic}} \\{location}\end{pmatrix}}{\begin{pmatrix}{{Aggregate}\mspace{14mu} {number}\mspace{14mu} {of}\mspace{14mu} {user}\mspace{14mu} {accounts}\mspace{14mu} {associated}\mspace{14mu} {{wit}h}\mspace{14mu} {the}} \\{{user}\mspace{14mu} {accounts}\mspace{14mu} {that}\mspace{14mu} {have}\mspace{14mu} {posted}\mspace{14mu} {from}\mspace{14mu} {the}\mspace{14mu} {set}\mspace{14mu} {of}} \\{{geographic}\mspace{20mu} {locations}}\end{pmatrix}}}}$

For instance, in the example above, the stadium has an I_(expected)=0.1.Thus, it is expected that the social influence (reach) of the stadiumwill be 10% of the influence that a set of stadiums has (i.e., thatI_(influence) will also be equal to 0.1). However, if the aggregatenumber of users reachable by the stadium is 20% of all the usersreachable by the set of stadiums (that includes the stadium), thenI_(influence)=0.2. Thus, according to the expression above, the stadiumwould have an SII=(0.2)/(0.1)=2. In some examples, an SII of greaterthan 1 indicates a higher than expected amount of social influence(reach) by a geographic location.

FIG. 23 depicts an exemplary process 2300 for accessing informationrelated to a hotel. As described above with respect to FIGS. 21 and 22,an exemplary geographic location is a hotel, and an exemplary set ofgeographic locations is a set of hotels. In some embodiments, the systemcan perform one or all of the steps of process 2300 in conjunction withperforming either or both of processes 2100 and 2200.

At block 2302, the system accesses data representing the number of roomsin a hotel. In some embodiments, this data is stored locally (e.g., atthe system). In some embodiments, this data is stored remotely (e.g.,over a wide area network connection). For example, the data can beaccessed from sources available on the Internet, such as hotel websites,travel websites, public records, or the like. In some embodiments, thisdata is stored locally, and refreshed periodically from one or more ofthe sources above.

At block 2304, the system accesses data representing the number of roomsin a set of hotels. In some embodiments, this data is stored locally(e.g., at the system). In some embodiments, this data is stored remotely(e.g., over a wide area network connection). For example, the data canbe accessed from sources available on the Internet, such as hotelwebsites, travel websites, public records, or the like. In someembodiments, this data is stored locally, and refreshed periodicallyfrom one or more of the sources above.

FIG. 24 depicts an exemplary process 2400 for accessing informationrelated to a venue with seats. As described above with respect to FIGS.21 and 22, an exemplary geographic location is a venue with seats (e.g.,a stadium, arena, or the like), and an exemplary set of geographiclocations is a set of venues. In some embodiments, the system canperform one or all of the steps of process 2400 in conjunction withperforming either or both of processes 2100 and 2200.

At block 2402, the system accesses data representing the number of seatsat a venue. In some embodiments, this data is stored locally (e.g., atthe system). In some embodiments, this data is stored remotely (e.g.,over a wide area network connection). For example, the data can beaccessed from sources available on the Internet, such as venue websites,ticket sales websites, public records, or the like. In some embodiments,this data is stored locally, and refreshed periodically from one or moreof the sources above.

At block 2404, the system accesses data representing the number of seatsin a set of venues. In some embodiments, this data is stored locally(e.g., at the system). In some embodiments, this data is stored remotely(e.g., over a wide area network connection). For example, the data canbe accessed from sources available on the Internet, such as venuewebsites, ticket sales websites, public records, or the like. In someembodiments, this data is stored locally, and refreshed periodicallyfrom one or more of the sources above.

d. Generating Reports Using Social Media Post Data

FIGS. 25A-25G depict exemplary representations of analyzed geographiclocation data as described above. The representations include conceptsdiscussed above, as well as other uses of location data, intended to bewithin the scope of this disclosure. In some examples, the systemdetermines one or more of the outputs of the data analysis conceptsdescribed above and below, and presents the results as a report.

In some embodiments, a report includes one or more of the portionsdepicted in FIGS. 25A-25G. In some embodiments, the report (e.g., as agraphical user interface) is provided to a requester (e.g., to a venue,via a venue computer system 110 of FIG. 1). In some examples, systemprovides the report to the venue computer system via an API, a website,or any other appropriate method for exchanging information betweencomputer systems. One skilled in the art would recognize that variousmethods for providing the information depicted in FIGS. 25A-25G (e.g.,graphics and/or text). In some examples, the report is a web portal. Insome examples, the report is a web-based application.

FIG. 25A includes information graphic 2500, which depicts a totalpotential reach (in number of users) (e.g., 121,000,000), a total numberof guests posting (22,000), and a total number of posts (32,000). Forexample, the total potential reach can be a value equal to the numberusers posting from one or more properties (geographic locations), aswell as followers of those users. The total number of guests (e.g., useraccounts) posting can be the number of guests at the one or moreproperties that have created social media posts at (or related to) ageographic location of the one or more properties. The total number ofposts can be the number of posts posted from (or related to) the one ormore properties.

FIG. 25A also includes information graphic 2502, which depicts socialmedia activity by region. In this example, if the one or more propertiesare spread out around the world (e.g., a network of hotels), then theoperator of the properties can view social activity broken down bygeographic region. Information graphic 2504 of FIG. 25A similarlydepicts social media activity by company (e.g., by subsidiary of theoperator), which can assist the operator in quantifying the userengagement with various business segments.

FIG. 25B includes information graphic 2506, which depicts a map andtable showing a Regional Influence Index (“RII”) for a variety ofregions. An RII is similar to the SII described above, but is a measureof the social reach (e.g., users and their followers) on a regionalbasis. In other words, the RII compares the percentage of reachgenerated from a region to the corresponding percentage of globalcapacity (e.g., total number of hotel rooms globally, from all hotels orjust those operated by the operator).

FIG. 25C includes information graphic 2508, which depicts popular topicsand words that have been identified from social media activity thatoriginated from the geographic locations of two groups of properties,Properties A and Properties B. Information such as that shown in graphic2508 provides analytics that are specific to the geographic locations ofspecific properties. Such granular information is extremely valuablebecause, for example, the operator can tailor social media engagementand other customer-facing activity based on the data, which wouldotherwise be unavailable. For example, if a particular topic (e.g.,Miami Festival 2018) is trending among guests at a single property, theoperator can see this information and respond (e.g., by engaging theuser on social media to provide a coupon for use at the Festival).Thereby the user's experience is improved (e.g., automatically) and theoperator's goodwill increased.

FIG. 25D includes information graphic 2510, which depicts a collectionof individual geographic locations (e.g., hotels), and the number ofsocial media posts that are being generated from each. As describedabove, this provides an operator the ability to view the aggregateamount of social activity on a property-by-property basis.

FIG. 25E includes information graphic 2512, which depicts a plurality ofsocial media user accounts and their number of followers. In thisexample, these user accounts represent users with a large amount ofinfluence (e.g., followers) that are posting from geographic locations.A property operator can engage with identified user accounts of highinfluence in order to maximize opportunities to engage with the largernetwork of the followers of these accounts.

FIG. 25F includes information graphic 2514, which depicts a graphicalview of the Social Activity Indices of various geographical locations(e.g., hotels A through Z).

FIG. 25G includes information graphic 2516, which depicts a graphicalview (resembling an iceberg) of the percentage of social media postswith identifiable hashtags and mentions (4% in this example) and thepercentage of posts with no identifiable hashtags or mentions (96% inthis example) for an exemplary set of data. As shown, the vast majority(e.g., 96%) of social media posts can otherwise go unnoticed byoperators of geographic locations such as venues, hotels, stadiums, andthe like. Location resolution, and the techniques related theretodescribed above, can be used to create and provide useful data thatwould otherwise not exist.

Attention is now directed to FIG. 26, which illustrates exemplarycomputing system 2600 which can be used as a computer system for a venueand to carry out, for example, any of the processes described above anddepicted in FIGS. 2 and 9-24. In addition, computing system 2600 can beused to facilitate the provision of the exemplary user interfaces anddatasets described above with respect to FIGS. 3-8. Further, computingsystem 2600 can be used as any of the devices, servers, or systemsdepicted in network diagram 100 (FIG. 1).

Exemplary computing system 2600 includes a motherboard 2602 having I/Osection 2606, one or more central processing units (CPU) 2608, andmemory section 2610. Memory section 2610 can be based on various memorymodules, such as DIMM memory modules. Memory section 2610 also can beoperatively coupled, directly or indirectly through I/O section 2606,with other memory modules, such as flash memory card 2612, a USB memorystick, and the like. I/O section 2606 is operatively coupled withdisplay 2624, human input device 2614, network interface 2622, and datastorage unit 2616. Data storage unit 2616 can be a disk drive,solid-state storage device, internet-based (e.g., cloud) storage, andthe like. Network interface 2622 permits computing system 2600 tocommunicate with a computing device, system, and or server such as thosedepicted in FIG. 1 and discussed above. Computing system 2600 canexclude one or more of the components listed above, and can includeother components not depicted.

Computing system 2600 can have computer-executable instructions forperforming the above-described techniques, including the processesdescribed above and depicted in FIGS. 2 and 9-24. Suchcomputer-executable instructions may be stored in memory section 2610.Memory section 2610 may obtain the computer-executable instructions fromvarious sources including flash memory 2612, data storage unit 2616,computer-readable medium 2620, network interface 2622, and so forth.Data storage unit 2616 may itself be, or may be a device configured toread from, a non-transitory computer-readable medium 2620 that is usedto store (e.g., tangibly embody) one or more computer programs forperforming the above-described techniques and processes. The computerprogram may be written using technologies such as C, Java, JavaScript,HTML5, Python, PHP, MySQL, Android software toolkit (“STK”) made byGoogle Inc. of Mountain View, Calif., and/or iOS software developmenttoolkit made by Apple Inc. of Cupertino, Calif., or the like.

Aspects of the embodiments disclosed above can be combined in othercombinations to form additional embodiments. Accordingly, all suchmodifications are intended to be included within the scope of thistechnology.

1. (canceled)
 2. A non-transitory computer-readable storage mediumstoring one or more programs configured to be executed by one or moreprocessors of a system, the one or more programs including instructionsfor: accessing data representing a first social media post that isposted by a first user account, wherein the data representing the firstsocial media post includes geographic location data identifying a firstgeographic location; determining that a second social media post isrelated to the first social media post, wherein the second social mediapost is posted by a second user account that is different from the firstuser account, and wherein the entity associated with the second useraccount is different than the entity associated with the first useraccount; accessing data representing the second social media post,wherein the data representing the second social media post does notinclude geographic location data identifying the first geographiclocation; analyzing the data representing the second social media post;determining a location score based at least in part on the analysis ofthe data representing the second social media post; and if the locationscore for the data representing the second social media post exceeds athreshold location score, associating the second social media post withthe first geographic location.
 3. The non-transitory computer-readablestorage medium according to claim 2, wherein determining that the secondsocial media post is related to the first social media post includes:determining that the second user account is mentioned or tagged in thefirst social media post; and identifying one or more social media postsposted by the second user account, wherein the one or more social mediaposts posted by the second user account includes the second social mediapost.
 4. The non-transitory computer-readable storage medium accordingto claim 2, wherein determining that the second social media post isrelated to the first social media post includes: determining that thefirst social media post includes a first tag; and identifying the secondsocial media post based on the first tag, wherein the second socialmedia post includes the first tag.
 5. The non-transitorycomputer-readable storage medium according to claim 2, whereindetermining that the second social media post is related to the firstsocial media post includes: determining that the second user accountinteracted with the first social media post; and identifying one or moresocial media posts posted by the second user account, wherein the one ormore social media posts posted by the second user account includes thesecond social media post.
 6. The non-transitory computer-readablestorage medium according to claim 2, wherein determining that the secondsocial media post is related to the first social media post includes:accessing a database of identifiers associated with users previouslyidentified as likely to be located at the first geographic location;matching the second user account to an identifier in the database; andidentifying one or more social media posts posted by the second useraccount.
 7. The non-transitory computer-readable storage mediumaccording to claim 2, wherein analyzing the data representing the secondsocial media post comprises: performing, by one or more processors,semantic analysis to determine a semantic similarity score between thesecond social media post and a collection of data representing socialmedia posts identified as being associated with the first geographiclocation, wherein the location score is determined based at least inpart on the semantic similarity score.
 8. The non-transitorycomputer-readable storage medium according to claim 2, wherein the datarepresenting the second social media post includes an image, and whereinanalyzing the data representing the second social media post comprises:applying, by one or more processors, a computer vision classificationalgorithm to the image of the data representing the second social mediapost to determine a first class confidence score; if the first classconfidence score for the image of the data representing the secondsocial media post exceeds a threshold first class confidence score,classifying the image of the second social media post as matching thefirst class, wherein the location score is determined based at least inpart on the classification of the image of the second social media postas matching the first class; and if the first class confidence score forthe image of the data representing the second social media post does notexceed the threshold first class confidence score, forgoing classifyingthe image of the second social media post as matching the first class.9. A computer-implemented method for resolving geographic location ofsocial media posts to determine if they are associated with a firstgeographic location, the method comprising: accessing data representinga first social media post that is posted by a first user account,wherein the data representing the first social media post includesgeographic location data identifying a first geographic location;determining that a second social media post is related to the firstsocial media post, wherein the second social media post is posted by asecond user account that is different from the first user account, andwherein the entity associated with the second user account is differentthan the entity associated with the first user account; accessing datarepresenting the second social media post, wherein the data representingthe second social media post does not include geographic location dataidentifying the first geographic location; analyzing the datarepresenting the second social media post; determining a location scorebased at least in part on the analysis of the data representing thesecond social media post; and if the location score for the datarepresenting the second social media post exceeds a threshold locationscore, associating the second social media post with the firstgeographic location.
 10. The method according to claim 9, whereindetermining that the second social media post is related to the firstsocial media post includes: determining that the second user account ismentioned or tagged in the first social media post; and identifying oneor more social media posts posted by the second user account, whereinthe one or more social media posts posted by the second user accountincludes the second social media post.
 11. The method according to claim9, wherein determining that the second social media post is related tothe first social media post includes: determining that the first socialmedia post includes a first tag; and identifying the second social mediapost based on the first tag, wherein the second social media postincludes the first tag.
 12. The method according to claim 9, whereindetermining that the second social media post is related to the firstsocial media post includes: determining that the second user accountinteracted with the first social media post; and identifying one or moresocial media posts posted by the second user account, wherein the one ormore social media posts posted by the second user account includes thesecond social media post.
 13. The method according to claim 9, whereindetermining that the second social media post is related to the firstsocial media post includes: accessing a database of identifiersassociated with users previously identified as likely to be located atthe first geographic location; matching the second user account to anidentifier in the database; and identifying one or more social mediaposts posted by the second user account.
 14. The method according toclaim 9, wherein analyzing the data representing the second social mediapost comprises: performing, by one or more processors, semantic analysisto determine a semantic similarity score between the second social mediapost and a collection of data representing social media posts identifiedas being associated with the first geographic location, wherein thelocation score is determined based at least in part on the semanticsimilarity score.
 15. The method according to claim 9, wherein the datarepresenting the second social media post includes an image, and whereinanalyzing the data representing the second social media post comprises:applying, by one or more processors, a computer vision classificationalgorithm to the image of the data representing the second social mediapost to determine a first class confidence score; if the first classconfidence score for the image of the data representing the secondsocial media post exceeds a threshold first class confidence score,classifying the image of the second social media post as matching thefirst class, wherein the location score is determined based at least inpart on the classification of the image of the second social media postas matching the first class; and if the first class confidence score forthe image of the data representing the second social media post does notexceed the threshold first class confidence score, forgoing classifyingthe image of the second social media post as matching the first class.16. A system, comprising: one or more processors; memory; and one ormore programs, wherein the one or more programs are stored in the memoryand configured to be executed by the one or more processors, the one ormore programs including instructions for: accessing data representing afirst social media post that is posted by a first user account, whereinthe data representing the first social media post includes geographiclocation data identifying a first geographic location; determining thata second social media post is related to the first social media post,wherein the second social media post is posted by a second user accountthat is different from the first user account, and wherein the entityassociated with the second user account is different than the entityassociated with the first user account; accessing data representing thesecond social media post, wherein the data representing the secondsocial media post does not include geographic location data identifyingthe first geographic location; analyzing the data representing thesecond social media post; determining a location score based at least inpart on the analysis of the data representing the second social mediapost; and if the location score for the data representing the secondsocial media post exceeds a threshold location score, associating thesecond social media post with the first geographic location.
 17. Thesystem according to claim 16, wherein determining that the second socialmedia post is related to the first social media post includes:determining that the second user account is mentioned or tagged in thefirst social media post; and identifying one or more social media postsposted by the second user account, wherein the one or more social mediaposts posted by the second user account includes the second social mediapost.
 18. The system according to claim 16, wherein determining that thesecond social media post is related to the first social media postincludes: determining that the first social media post includes a firsttag; and identifying the second social media post based on the firsttag, wherein the second social media post includes the first tag. 19.The system according to claim 16, wherein determining that the secondsocial media post is related to the first social media post includes:determining that the second user account interacted with the firstsocial media post; and identifying one or more social media posts postedby the second user account, wherein the one or more social media postsposted by the second user account includes the second social media post.20. The system according to claim 16, wherein determining that thesecond social media post is related to the first social media postincludes: accessing a database of identifiers associated with userspreviously identified as likely to be located at the first geographiclocation; matching the second user account to an identifier in thedatabase; and identifying one or more social media posts posted by thesecond user account.
 21. The system according to claim 16, whereinanalyzing the data representing the second social media post comprises:performing, by one or more processors, semantic analysis to determine asemantic similarity score between the second social media post and acollection of data representing social media posts identified as beingassociated with the first geographic location, wherein the locationscore is determined based at least in part on the semantic similarityscore.
 22. The system according to claim 16, wherein the datarepresenting the second social media post includes an image, and whereinanalyzing the data representing the second social media post comprises:applying, by one or more processors, a computer vision classificationalgorithm to the image of the data representing the second social mediapost to determine a first class confidence score; if the first classconfidence score for the image of the data representing the secondsocial media post exceeds a threshold first class confidence score,classifying the image of the second social media post as matching thefirst class, wherein the location score is determined based at least inpart on the classification of the image of the second social media postas matching the first class; and if the first class confidence score forthe image of the data representing the second social media post does notexceed the threshold first class confidence score, forgoing classifyingthe image of the second social media post as matching the first class.